Planning to customize your new cloud-based system via the APIs? Proceed carefully..

In my work with libraries, I’m increasingly hearing and being asked about concerns with the new “open” systems and their Application Programming Interfaces (API’s) which many libraries use to meet local needs.  I’m afraid these concerns are only going to increase as more of the profession moves toward cloud-computing or hosted solutions. Specifically, the concerns I’m hearing are:

1. A lack of adequate or usable API documentation.   I’ve been working with automation systems for libraries since the early 80’s and complaints about software documentation are as constant as the rising and setting of the Sun.  These complaints aren’t without reason.   Many libraries will, as a result of the move to cloud-computing systems, be more likely to use APIs as a way of providing needed customizations for their library members.  This requires greatly increased dependence on API documentation and it’s overall usability.    Library staff APIs from suppliers that are difficult to use, require training to use or that are overly complex are, to put it bluntly, examples of poor programming.  Organizations developing these APIs should have internal standards that have to be met and their Quality Control group should be checking them in order to avoid these problems.  Ultimately, if this isn’t done, your library is going to absorb a substantial cost overhead in lost development time, not to mention staff frustration, that no library can afford in this economy.  Solution?  Before signing on to buy or upgrade to a new proprietary system, demand to see the API documentation and have the supplying organization give you access to an API of your choice.  Ask one of your staff to do some test development using that API.  If s/he comes back to your desk babbling and looking crazed, you’ll know to look at other systems.  If your person comes back smiling and eager to show you what they’ve done, you’re on the path to the right solution.
2. The API/Code-extension developer community doesn’t seem to really be a community. Many organizations and their customers, in moving to what they’re calling “open systems”, are engaging in more marketing than real change.  While a proprietary vendor may be quick to adopt the broader terminology of the open source community, including the word “community”, there may not be substance behind that adoption. OSS developer communities understand that part of the obligation they take on as contributors and developers is the need to help provide support for new users entering the community or to help those trying to implement or work with developed code.  These communities have their own codes of conduct and recognition and the bottom line is that they usually work well.  You can get support out of a well-run OSS community behind a product.   However, when suppliers of proprietary systems adopt the phrase “developer community” it may not mean the same thing.  They aren’t bound to the same rules as the OSS community. Frequently the code isn’t as openly available because the vendor won’t allow that to happen (check - is it on SourceForge?) or there might be legal restrictions in the contract covering the use of the code contributions, especially if the vendor has supplied them.   Code contributions from customers may not be actively supported by those customers due to other demands on their time.   Or they may feel that the vendor should do this or at least, should not profit in any way from the support they provide. The result is that the support may or may not be anywhere near as good as you’d find in a total OSS community.  Solution?  Check to see if the vendor/supplier offers support for code extensions either on a pay-as-needed basis or as part of a maintenance contract.  Be aware that most will not as their resources are dedicated to other projects, sometimes for years in advance. See if the vendor/supplier will let you examine support requests that have been placed on a listserv or through other communication vehicles they provide.  Verify what the typical response time is and how often such requests go altogether unanswered.  Look at the resulting data and make a fact-based judgment of the results to determine if support is really available and if it will meet your needs.      
3. No pre-notification of upcoming changes to API functionality.  In the case of a locally installed proprietary system a library can wait to upgrade until they’ve rewritten their code extensions to work with the newly revised and delivered API’s.  However, with the increasing move towards cloud computing or hosted proprietary solutions, that control is moving to the hands of the vendor/supplier.  This offers the potential for far more problems.  Solution?  Libraries should, before buying into one of the new cloud-based proprietary solutions, check to see if people from the user community are involved in reviewing API changes before they’re made.  Libraries should also insert into their hosting contracts, language that specifies at least a 90-day notification of changes to the vendor API’s that would affect existing API functionality.  Furthermore, the documentation should be supplied at that same time so that libraries can prepare their code extensions to use it. If the changes are adding new functionality, then the notification is not as critical, although it would still be a good business practice.
4. Contract restrictions in API usage are unspecified, unclear or can be modified without notification. I understand that many supplier organizations are particularly sensitive to the issue of their APIs being used by their competitors in developing new products or services that will end up competing with the system suppliers offerings.  Yet, I’m not convinced libraries should simply agree to this.  Librarians should have the right to pick from the best available solutions, whoever develops them.  Of course, suppliers/vendors will want to leave usage terms deliberately vague so they can define them as needed.  They might want to restrict usage to non-profit organizations (or if they fear OCLC, they may try to restrict usage to your library staff).  I’m not saying this is right but at least you can understand their desire not to make it easy for their competitors. In some cases, even if the library is the developer, a vendor/supplier might attempt to stop them from developing a competing offering, particularly if the library intends to place the resulting code in open source (that situation is really NOT defensible in my opinion, but it does happen).   It is also the case that many contracts may contain API usage clauses that allow the contractual terms of usage to be modified by the supplier without the consent of the customer.  This is dangerous for libraries and I would advocate it shouldn’t be agreed to under any circumstance.  Solution? Before signing any agreement covering API utilization, make sure the terms are clearly spelled out and that you retain the right to develop whatever applications you wish.  Also, put it in writing that you can share the code you develop with other similar organizations.  If you truly want to foster greater innovation in the market, demand the right for all types of organizations to develop software that use these APIs.  (Note, it would be fair for the vendor to require that those organizations also be supplier of software or services to your organization).  You’ll likely have to fight for this one, but depending on how strongly you feel about innovation and getting the greatest return on your investments, the fight might be worth having.  Finally make sure all these legal terms can’t be modified without your written consent. 
5. The need for standardized API’s.  Depending on the system involved, for example a library management system, you might use the system anywhere from 3-10 years.  Some systems, like discovery or ERM systems might be in place for a shorter time frame.  In virtually all instances and at some point down the road, you’ll likely want to switch to a different supplier/product.   At that time, having the capability to switch all the code extensions you’ve developed from your existing system to the new system will be dependent on one thing – the existence of standardized APIs.   Most vendor/suppliers want to ignore this request or place it into a standards development organization.  There they know the resulting standard will likely be a  “lowest common denominator” because libraries don’t participate in the standards process at a level that allows them to prevent this from happening.  But this is one of those places where libraries need to take hold of their future and define what it is going to look like.  Particularly with the move to cloud based systems.   Solution?  Libraries should insist on standardized API interfaces.  They should join and participate in NISO or other organizations where the standards work is being done to ensure the resulting standards support true transportability of extensions between systems.   If you don’t do this you’re setting yourself up to have to do all the development work over (for no good reason) and you’re penalizing your library users/members with loss of functionality while that happens. 
6. You’re locking yourself to a vendor/supplier and the course they choose.  I’ve written before about “The disintegration and redistribution of the library”.  If you’re not careful with your usage of APIs, you might be aiding and abetting that process.  Think about what would happen if you haven’t done some of what I’ve suggested above and a different organization buys your supplier/vendor?  Or, where would you be if an entirely new administration moves into your suppliers operation?  What if they shut down access to those APIs, substantially modify them or withdraw support altogether?  What if they decide they want to serve your library user/members direct and decide to cripple your library by withdrawing your ability to use these APIs?   Be particularly thoughtful about this when you’re buying your discovery interface from the same firms that supply your content.  I keep pointing out that libraries don’t overall represent a growing market and in most cases libraries are dealing with shrinking budgets.  This makes for a difficult market for vendors/suppliers who are facing owners/shareholders that want to see continual and substantial growth.   As a consequence, they will be forced to look for new, adjacent markets or ways to grow beyond simply serving libraries and one way to do that is go around the libraries.  Think it can’t happen?  I suggest you look at firms like Pearson, McGraw-Hill and others where courses are being bundled with extensive content and offered directly to university provosts, department heads, professors, etc.  All without a library in sight.  Then also consider what is happening with e-books at the moment. So, yes - you bet it can happen. If you’re not careful, you might be signing a contract that will place your library in this position.  Solution?  Get advice, do your due diligence, read relevant background materials and proceed quickly but carefully.  

Ultimately, I still hope we see the day when APIs will largely be replaced and/or supplemented with much easier customization tools, particularly for the cloud-based systems.  For instance, I’m constantly amazed at how simple it is in Google Blogger, to add functionality and totally redo the interface or product functionality by using simple drag-and-drop widgets.   That kind of customization is what librarians should be demanding from their suppliers.  It’s not news that many libraries can’t afford to have or retain good technical programming staff.  The fact that they’re potentially being penalized further by the inability to easily customize their systems in order to meet end-user/member needs adds is not good.  Libraries should join together and speak loudly with a common voice and through a common organization to state their needs with regard to API usage so that these concerns can be addressed.  


(Note: - Want more advice for your library?  Contact me at the CARE Affiliates website)

Understanding context like we've never understood it before

I've long considered the CNI Membership Meetings one of the most important events on my calendar because I rarely leave them without having a serious "wow" moment.  Such was the most recent meeting in Baltimore during the closing plenary session by Phillip Long of the University of Queensland.  In a few brief moments within the talk about "Key Trends in Teaching & Learning" we were shown some of the research that is happening at the Gallant Laboratory at UC Berkeley.  There they have "modeled brain activity elicited by static visual patterns and have reconstructed these patterns from brain activity" and taken it to the next step" by overcoming some earlier limitations in this research.  What this means is that they've now come to the point where they can "reveal how early visual areas (in the brain) represent the information in movies."   They can now decode what the mind recorded.  This is incredible to watch (there are some additional sample videos at the the website that show what is recorded by the brain and, while some are vague and unfocused, some are reasonably sharp.  Plus, as Phillip Long pointed out: It is early in the process so this will only get better).  

It appears that what the research is showing is what the mind records, not  what was actually shown/seen and therefore is subject to all the twists and turns that humans apply to memory.   That is a fascinating study when done, all by itself (the classic case of five people seeing an event, then recounting it, and all five having variances).    However, the questions that result are major:  Are those memories there even when we can't consciously recall them? Can these be recalled whether you want them to be, or not?  (Think about the implications of that one...).   It raises all kinds of issues ranging from legality to ethics to human-machine interfaces to what it will mean for learning.  

However, as librarians this gives a whole new meaning to providing knowledge with context; a huge differentiator for librarianship when compared to other information and knowledge access  services.   As librarians, we often struggle to make sure library users analyze information and knowledge within the context of the time and knowledge in which it was created.  To now be able to see how visual information is recorded in the mind and to ensure that the context is properly provided and understood will give us some incredible insight into how to provide access to information and knowledge.   (Of course the potential for abuse here is large and will require major new ethical boundaries to be defined and implemented to go with this insight).   

I've discussed information filtering and context and the issues surrounding it in a previous post and those remain valid in this visual environment.  However, what I saw at this CNI meeting means that as a profession, librarianship will need to stay on top of this development because the tentacles of this research reach deep into our work and will have major implications for our future both as providers of access to knowledge but also as participants in the creation of new knowledge.   

Steps being taken towards the new bibliographic framework too narrowly focused?

This week the Library of Congress, issued a survey as a next step in their continuing efforts to define the next generation bibliographic framework.  The cover message states:  

The survey is: “..studying the value and use of Library of Congress' bibliographic data and cataloging products. The.. survey results as part of a strategic study to guide its response to this changing environment, supporting the Library in its goal to effectively define its future role, adapt a sustainable financial model, and better serve its mission in the years ahead.” 

"The survey is directed at:  1. managers of cataloging and technical services units 2. catalogers 3. vendors and distributors of bibliographic data and tools”

I’ll admit that in reading this, I continue to worry the focus is too narrow.  I know that Karen Coyle has posted on this previously and again yesterday. Todd Carpenter of NISO has also noted the need for more participants, as have I in a previous post.  


While I think all agree what the Library of Congress is doing is very important and we applaud them for doing it, there remains a concern for all to remember this is important because it represents the chance to define the overall future usability of library bibliographic data across the entire Web.   


To truly enable that, we need to make sure we’re directing requests for input to communities like the Dublin Core Metadata Initiative (DCMI), and the World Wide Web Consortium (W3C), as well as a broad number of national libraries from across the international community.  In LC’s original announcement, they noted: 

“The Library of Congress’s process will be fully collaborative.  We will consult our partners and customers in the metadata community, standards experts in and out of libraries, and designers and builders of systems that make use of library metadata.”

If you look at this survey, there truly seems to be some inconsistency between what the statement above says and what this survey is asking (and to be honest, I’m having a hard time trying to figure out exactly what they DO intend to derive from the questions being asked?).   It certainly does not appear to be the case that they want to hear from: a) all the parties interested in using the data or, b) to provide an open dialogue that one would think is needed for a deep understanding of all the possible user needs?  If so, one must question this process.


Like many of you, I fully understand the mission of LC is “..to support the Congress in fulfilling its constitutional duties and to further the progress of knowledge and creativity for the benefit of the American people.”  Understood.  However, the Library of Congress does function, and clearly wants to function, as our de-facto national library.    For instance, note the word “and” in that mission statement when they’re talking about furthering knowledge and creativity for the American people. So if they want to further knowledge and creativity, they have to work towards making library data cleanly and easily integrated across the rapidly unfolding Semantic Web.    


It is also painfully apparent in this call to answer a survey that we’re not being given a clear indication of how this input will factor into the final decisions.   Surveys can have a significant impact on the results or none at all, depending on the weight given the results in the total decision making process.  That weighting needs to be shared with all who are responding so they’ll know if it is even worth their time to respond.


Personally, I believe this very important step deserves wide input from the all the communities I (and others) have mentioned and it needs to be conducted in a way that provides open discussion (in other words, some face-to-face discussions).  The end result should be a clear understanding of the needs of the primary providers and users of library bibliographic data.  Yes, this will cost more to do and it will probably require a number of interested parties to step up and help provide that funding, because I’m sure our Congress will also be too introspectively focused to see or understand the larger needs at play here.  However, it would seem to be a very wise investment with a very good return.


Along this line, I was very encouraged earlier today when I saw that one of the upcoming European Library Automation Group (ELAG) sessions is by Lukas Koster on Linked Library Data in the New Bibliographic Framework  and that they intend to provide a copy of the results to LC.  Bravo!  This is precisely the kind of input LC needs but seems reluctant to gather. Of course, that doesn't mean LC will weigh or use the input.  If that remains the case, then I agree with the closing remarks in Karen’s blog post where she says: 

“If we don't step up to this task, for many years to come we will continue to see library data housed in frameworks and silos that are invisible to most information seekers.  That would indeed be very unfortunate.”  

Very, very unfortunate.