David Clover 18 October 2013 09:37:40 AM
Our students will soon have to submit all their Electronic Tutor Marked Assignments
(ETMAs) in a single PDF file for marking, but there is a problem for those, for example in the Maths and Statistics areas and others, whose ETMAs have to consist of a heterogenous variety of original file types - .The file formats that we can convert to pdf and then merge include: doc, docx, pdf, jpg", "jp2", "jpf", "gif", "png", "tiff", "tif" objects in various formats. The new requirement is for these to be presented for assessment as a single concatenated PDF file.
My colleague Henryk
is putting the finishing touches to an Apache Tika
processing tool to detect content types which then feeds into a utility from Qoppa software
driven by Java via our Domino server. Students will be able to submit a .ZIP file containing all the disparate objects which are uploaded and after Tika's filetype detection, processed by Qoppa into a single .PDF file which students can then submit in the usual way.
We have spent some time attempting to develop an IBM Domino XPages tool for this, but because of incompatibilities in processing .docx files between the Qoppa back-end environment that does the processing and the XPages environment we have had to re-think and have returned to 'conventional' web input techniques using LotusScript and Formulas. Having now solved some Java runtime permission issues on the server, this has now been shown to work extremely well, and so it just remains to build and tune the web front end in order to deliver the system to our students.
The workflow concept is that a student will authenticate with the web page as usual, and then upload the ZIP file. After a short wait, during which the protocol conversions are done on the server in the background, he or she will be presented with a web page with the uploaded .ZIP and the newly concatenated .PDF file. This can then be downloaded and passed across to the University's ETMA system in the usual and approved way.
David Clover 12 September 2013 10:10:38 AM
A neighbouring department had an urgent need for a utility to meet a work planning need. The office supports a wide range of activities across the organisation. To date, the office had been using a shared MS Excel spreadsheet to manage work scheduling, and this approach had been used for several years despite a number of disadvantages – only one person could use the spreadsheet at a time, records could be accidentally over-written or deleted with no audit trail and the amount of data that could be displayed was limited. Excel is not a suitable tool for a task of this complexity. The schedule also employed a complex system of colour coding to indicate priority, status and due date, and these colours all needed to be applied by hand where relevant: a time-consuming manual task. Despite these limitations, this Excel schedule was the main record of ongoing, upcoming and completed work for the whole survey office team and as a result was integral to the office’s smooth running.
While moving to a web-based tool for managing their scheduling has been a desirable idea for some time, it became a higher priority following an incident where the office found that a record had been accidentally deleted, causing inconvenience to a client who was expecting some results and considerable disruption to the office as work had to be hastily re-arranged to prioritise work that had been missed.
So an IBM Domino web-based tool was created by a member of our MCT IT-Development Group running under XPages, and because of the urgency, it was developed in two days of intensive programming and analysis.
The main benefits of the new approach are:
- Everyone can use the schedule at the same time;
- It’s impossible to delete a record by accident;
- ‘Info’ entries (the purple bars across the whole sheet) will automatically disappear once their date has passed;
- The client group will be able to record more details about jobs than are currently possible – a notes field has been added which can hold an unlimited amount of text;
- Each member of the client group has a personal view so each person can see just those jobs that are relevant to him or her;
- People outside the survey office, including managers, can be given access to use the link to see jobs in progress without needing access to server folder shared area;
- The green/blue shading for this/next week’s jobs will be applied automatically over time rather than needing manual updating;
- Jobs will automatically move to the ‘finished’ tab when they have been completed.
The client department is thrilled with the new environment and it represents another significant achievement for the rapid development potential and flexibility of our Domino 9 XPages rapid development environment.
Here's a screen shot of what the user sees:
David Clover 9 September 2013 11:13:00 AM
For many years - since 2004 in fact - we have been showing 'personal' pro-forma details in the context of departmental websites using a back-end Domino database. But up to now, updating those details has only been possible with an IBM Notes client, and that's been restricted to those people responsible for content updating in various departments.
Now that is about to change for the Faculty with the imminent release of a web-based interface at http://mct.open.ac.uk/you
- this will allow a user to login and then see and edit their own personal details directly.
The information is stored in our tried and trusted IBM Domino database (which we call 'MaCuNames' - it drives a number of Faculty 'people' information systems) but the entry about a person can now be extracted and displayed on any web page served from the campus using a server-side XML request with a specifically tailored URL. Campus users can see such an extract for my own details and possibly their own by altering the URL slightly - David Clover
This means that we can make a server-side request from our Drupal environment to populate a page like my own
in the new Faculty website, and we will also be able to populate any other site that can make a similar server-side call to present the same data in whatever context is needed. We have designed it so that it is backwards compatible with older websites which use IBM Domino as the presentation engine without any changes.
As far as editing the personal content is concerned, we provide three 'slots' for users to populate. These are 'Biography
' and 'Teaching
'. If a particular 'slot' is not filled in, the related tab won't appear on the presented public web page. In my own example case I have completed all three, but in the case of others, for example my colleague Blaine Price
in the Computing and Communications Department only the 'Research
' tab is visible. The other tabs - 'News
' and 'Publications
' will appear if the person represented has any news items or research publications recorded against their name. 'News
' relates to our own 'News' module (also an IBM Domino back end source with a web-editing front end) and our 'Open Research Online
' database operated by the Open University library. It's also possible to email the person via a web form- which ensures that spam harvesters can't scan the page for an email address. The web form is protected with a 'Captcha' module. We've also included links to other 'Social' aspects for the user -including Facebook, Twitter, RSS feeds from blogs, personal web areas and so on.
Some of the information relating to a person is imported daily from the University's HR system, and some is held only in the Faculty. Users can modify the details directly where they are held in the back-end IBM Domino database, but where the information is imported from elsewhere, we provide outward links to the University's HR web-editing sites where they can modify their own data for overnight automated import to our own environment. We have also provided an option whereby people can opt not to have their details selected for websites at all if they want to remain anonymous.
There are plans afoot to provide a University-wide service for holding 'people profile' data but it's not likely to be ready for some time. When it is, we expect to be able to import from it to our own system (or indeed export from our system to it as the case may be) to provide a standard and coherent statement visible wherever it is needed on a website. Examples of the input screens are shown below.
David Clover 25 June 2013 11:38:59 PM
Our current and major task, and certainly not a trivial one, is to provide a new 'focal point' website to reflect major organisational changes that have just taken place in our Faculty of Mathematics, Computing and Technology
This had originally been planned for October 2013 but was brought forward to August 1st 2013 and so this gave us very little time to build the environment. In fact the services on the new PHP sites referred to below didn't exist a month or so ago.
The 'Presentation' layer uses a popular bulletin board PHP framework (Drupal), but large sections of it are populated dynamically using XML and Web Services from our existing IBM Domino content containers. These containers also service other existing and 'legacy' websites and services which are presented directly from the Domino http server - sites such as Centre for Research in Computing
and sites such as our Faculty intranet
We have described the plan in outline for (in-house readers)
in a PDF file.
We have been instructed that many of our team's (and indeed users') long-held assumptions about how we represent the Faculty and its three new departments via a web information system have to be put aside, so we are, to a certain extent, making it up as we go along.
What is clear though is that it is very easy to exploit the mature XML techniques available from our existing IBM Domino databases and infrastructure to provide fast and efficient web services so that we can capitalise on the data containers we already have for Staff and research projects, news and events. We can do this in a way that allows us to re-present single sources of data within the PHP environment via XML and linked data. For some examples see these pages:
We have done the same with news items at:
From a development point of view, it would have been very much simpler for the team to have devised everything within a single Domino data and presentation framework, but the Web Services, XML, Linked Data and SOA elements that we can expose from our back-end Domino containers are the key to flexibility in commercially focused organisations and we can use them too. We should be able to extend this seamlessly as needed, and it will extend greatly the technical range of our team which is now well-versed in the existing and simple to manage IBM Domino 9 environment and increasingly the PHP framework we have been instructed to use.
As ever with web-based environments, especially one created very quickly like this, the technical challenges are more than matched by the even larger challenge in getting meaningful content into the new site; we will probably be plundering our 816 or so existing Faculty sites
for suitable content in the short term.
We will offer staff working as editors and content providers with an Active Directory login via LDAP to the new site for editing and they will use 'in place' editing of static PHP content. We'll also be providing web-enabled editing of the dynamic/SOA content using Domino XPage interfaces such as http://mct.open.ac.uk/you
which we are just finishing now.
David Clover 19 June 2013 04:34:20 PM
Every year our hardworking Associate Lecturers carry out a large number of face-to-face tutorials all around the country. This is how we bring the 'local' element into the distance teaching we do as a national University specialising in Supported Open Learning
We need to collect statistics about how many students attended the face to face sessions, and to record for onward report to the University how many scripts were received from the group that was tutored.
Elsewhere in the University this troublesome chore is carried out in a very 'manual' way, perhaps by filling in and concatenating spreadsheets and netting them up laboriously into summary reports. However, for some years, our Faculty of Mathematics, Computing and Technology has been collecting this electronically using our IBM Domino server, and this year, we completely re-built the system using IBM Domino XPages. This automated collector is now up and running very successfully. So far we have collected nearly 2000 records from our Faculty's Associate Lecturers.
The IBM Domino system, developed by my colleague Jef, makes extensive use of XPage controls; an example is shown below. We also use a combination of Active Directory and Web SSL login cookie detection techniques to collect some of the details of the tutor completing the forms. This means that each sees only his or her own entries and only validated users can make entries and changes. Once they have authenticated, we can allow them to edit and delete their own entries too. Using Domino's useful ACL 'Roles' we can quickly assign or emove extra privileges to those who administer the system so that they can see all the entries from all users in the same web interface.
Another XPage control allows us to upload the entire collection of reports, or part of it to an Excel sheet. We can also show as tooltips in the list of items any comments that have been placed on the underlying document. You can see one of these in the sample below. A simple workflow routine checks that the entries placed on the form are complete and consistent and advises the user of anything that might be missing before saving,
Once again, IBM Domino and XPages has provided elegantly and quickly (in one easily managed Domino database object) a solution that would have taken much longer to develop using alternative techniques.
Fig 1: Log Display for a user
Fig 2: creating a new entry.
Fig 2: Offering an option to edit or delete an existing record
David Clover 26 April 2013 12:11:44 PM
We were asked to put together quickly a survey form which will be used in-house to collect details relating to 'External Events' requesting the levels of support required for them.
The survey form had a number of 'conditional' elements - some dialogue boxes were to appear only when certain checkbox options were chosen by the respondent. Some single questions had single 'radio button' choices and some single questions had a requirement for multiple check boxes to indicate preferences. The whole had to be secure and visible to a respondent only after logging in to our Campus authentication system (which we call SAMS) and it had to recognise the user to allow the form-filling to proceed.
My colleague, Jef Lay, assembled an suitable form within a day using standard XPages controls. The form will collect the data and a separate view just accessible to the requestor will show responses and allow download of the responses into an Excel format for subsequent manipulation.
There was no comparable option available at short notice and once again we've proved how efficient and versatile the Domino XPages model is when it comes to working with web-enabled environments.
Here's a snapshot of the form which is also visible at our Website to internal staff
David Clover 17 April 2013 11:16:07 AM
We are working with our Mathematics Programme area to develop a web-driven 'Assets' utility on which we can record all the web-based resources that we make available to students and enquirers.
We have a number of Maths-related websites which we need to keep track of. These also contain a number of web-delivered objects - PDF files, images, documents etc - that we want to 'single source' from a central point. When we update the resource we want them to be visible in the new version on all the related sites wherever they are.
My colleague Jacques has built a IBM Domino XPages system (his first!) to allow the Maths Programme administrative staff to enter and edit, via a web input form, all the details that make up the multitude of web-based information sources we have put out for public view. Jacques and Jef have worked out a way to give each PDF (or other reference object stored in the system for public consumption) a unique 'short code' which will work in perpetuity to avoid the problem of over-long machine-generated URLs.
We'll be using a full range of XPages controls and security techniques to allow the in-house users to search for items and read and define the context for each item. We can use the same database as the source for PDF and other objects that need to be referenced on other sites. These will be uploaded as needed to an 'asset' form. We'll also have utilities to 'find' items (we'll use a 'reducing' dynamic find technique built into XPages for this) and bring forward for action items which need reviewing at end-of-life or at the change of a presentation from one version to another.
We have set the security to allow authorised people read/write access, and we've used Domino's powerful IP filtering tools to allow in-house computer users to view and search the items whilst inhibiting external users from seeing it unless they are authorised as editors. This is very simple given IBM Domino's pre-existing and powerful range of security options and features which are fully accessible through XPages.
Jacques has had great success in manipulating the XPage controls for paging, and has managed to fix things so that after viewing an item the 'Home' button returns the user to the same point in the list that was used to call up an item. But he has had to use a customised 'pagerSaveState
' enhancement that he's devised which seems to work better than Domino's built-in one.
University staff can review our test read-only site.
For others, here are some screenshots of the work in progress.
David Clover 7 February 2013 09:15:28 PM
We've been reviewing an excellent IBM Domino 9 template from OpenNTF
which seems to offer a well-conceived, well-programmed and generally useful 'Social' utility which we can exploit to create small online 'communities' for research and other projects.
The look and feel of the 'Intrapages' environment
from Thomas Adrian
in Sweden owes a lot to existing public social networking environments - a strength in that the conventions people have already learned from using them in their private time are readily applied here. The template uses IBMs Domino 9 beta server which we have installed (it can also use Domino 8.5.3 with the OpenNTF Extension Library) and it is written using IBMs immensely powerful 'Xpages' Java-based programming environment.
It's a professionally-crafted and, so far robust, easy-to-understand interface. Here's an example:
The template offers 'Self Signup' - a distinct plus as it allows people external to the University to take part in the discussion - as well as offering an integrated Domino Directory sign up for organisations with users who reply on Domino for other workflow and mail-enabled applications. However, to meet all our needs, it will also need to offer Active Directory signon using LDAP. If that can be added to the design it has potential to become a seriously useful tool for us in providing services to researchers on projects. In fact we already created an Active Directory/LDAP login on our locally modified version of Declan Lynch's 'Large File Sender
' template a while ago so I hope it should be feasible to add it as an option here too.
Installing and configuring this powerful tool, a single .nsf database with no external dependencies, took all of 20 minutes from a standing start. Using Thomas Adrian's clear instructions, the parameterisation and tailoring of the interface was extremely easy to do. It offers the option of integrating logins with public networks such as Facebook and Linked in as well, though we shan't be doing that just yet.
I've been critical in the past of some of the offerings at OpenNTF which have promised much but then been incompletely finished, buggy or not maintained, but this one feels as though it shares the same build and programming qualities we found a long time ago in Steve Castledine's 'Domino Blog
' which is now released by IBM with its Domino server as an official 'core' template (indeed this blog uses it).
There are lots of organisations out there with a Domino infrastructure for whom this could become a really important tool in exploiting the significant power (and backend simplicity) of IBM's Domino infrastructure. Thomas Adrian has made a test environment available at http://www.intrapages.com/intra.nsf/start.xsp
where you can try it out.
David Clover 7 January 2013 04:01:51 PM
I've taken the plunge and replaced Windows 7 on my home machine with Windows 8 32-bit. It's an older AMD Athlon 64 processor 3700+ 2.21 Ghz, bought 16-Jun-2006. To make the Windows 8 system usable (and as initially installed, the Windows 8 'Modern' or 'Metro' interface simply isn't appropriate for a conventional desktop user) I've applied a few simple tweaks which have transformed the experience.
I could have installed the 64-bit version but wanted to compare what the 32-bit version did with the Windows 7 install. The installation did delete all the pre-existing programs on the machine, but kept all the personal settings and data so although it took a little while to reload everything, for the most part everything came back up properly including my MS Outlook Mailboxes and settings. It's sometimes worth starting from scratch to optimise the machine so that wasn't too much hardship.
Now that's done I have what feels like a fast, slick, and so far very robust, Windows 32-bit environment whose performance in all areas seems to surpass the previous Windows 7 build on exactly the the same hardware.
I very much like the improved approach to file backup (now called 'File History
') using Windows's 'Libraries', though I had to specifically include my drive E: files disk area into a new 'Library' to force it to use the new feature which now backs up to a 1Tb external drive. The video codecs seemed improved and there's now no 'jumping' when looking at MP4 HD video files from my camera.
'Taming' Windows 8
1. The first thing I did was to install the free Start W8 utility.
This useful piece of software has received a Microsoft Compatibility rating
and ensures that on starting, Windows 8 goes straight to the 'proper' Windows environment and not to the dreadful new 'Metro/Modern' interface. Everything you knew and loved about using Windows 7 works just like it used to. There's a wide choice of Start menu replacement utilities for Windows 8, but many seem to be over-complex for an ordinary user. However, 'Start W8'
was very simple, elegant and more than adequate.
2. Secondly I installed the 'God Mode' folder
which gives easy access to all the settings and configurations in Windows 8 from a single location. You'll rarely (if ever) need to struggle with Microsoft's nonsensical 'Charms' bar after configuring this.
I now have a fast booting machine which is available in less than 30 seconds (Windows 8 is very fast indeed to boot). I've been using Windows for 20 years and It feels very robust and efficient. I was pleased to find that the on-board network card that used to sulk and not work every time on start-up in Windows 7, fires up properly at every boot now. The only failure was a copy of IBM's Sametime 8.5.2 won't start - and IBM can't advise as it's not been tested on the platform yet. Fortunately I have a work around using IBM's new Notes 9 beta which performs flawlessly in Windows 8. I did however need to tweak the settings on my Cisco VPN installation
before it would work.
Now I rarely, if ever, have to visit the ill-conceived 'Metro/Modern' interface' other than to convince myself now and then that I still don't need it. I can use all my old software, hardware (scanners, cameras, printers etc) just as I always did. The apps supplied for the 'Metro/Modern' interface are poorly conceived and simply don't work well even in the single screen 'Microsoft Window' environment that they are nominally designed for. Jakob Neilsen's criticisms are well expressed. as are those of Phillip Greenspun
It's clear that Microsoft has taken a massive wrong turning in configuring Windows 8 for the desktop world. It's clear also, being fair, that the OS engineers have done an excellent job in bringing Windows 7 forward to version 8 as a reliable, improved and effective core platform. The wrong turning lies in the disastrous marketing decision (which I suspect many Microsoft insiders regret - especially the technically-savvy ones) to prioritise the absurd 'Metro/Modern' interface for the conventional desktop as well as for the tablet and phone.
Now I can safely regard the 'Metro/Modern' environment as an interesting but essentially non-intrusive 'sideshow' and get on productively with the things I need to do on a conventional multi-tasking Windows desktop just as I have been doing since the first version was released.
There's a long 'rant' by Brian Boyko on Windows 8
which is well worth a look. Everything Brian says about the 'Metro/Modern' interface is 100% true, but by carrying out the few tweaks mentioned above, I have safely bypassed all or most of the pain it represents and got myself a much better 'conventional' Windows desktop computer than I had before! The Windows 8 Survival Guide video
is also worth looking at, the same team's 'Windows 8: it's Almost Not Terrible
' which is less formal (and more fun).
David Clover 19 December 2012 01:43:19 PM
We are working on a very interesting and exciting JISC funded project which is concerned with the concept of 'Community Curation
Scanned Corpus of old books
Briefly, there is a large corpus of books and material printed in the 19th C and early 20th C which contain highly detailed and technical taxonomic descriptions of plants and animals in the natural world. Many of these books have been scanned recently by OCR. Now the original page images and a word by word breakdown of the OCR for each page is available to researchers. The problem is that like all OCR, there are mistakes in the scanning which prevent it from being used as a reliable and searchable resource for present and future researchers.
OCR Comparison - protoype tools
The ultimate aim is to submit the OCR'd version alongside the original page to experts in the bio-community. They will work through and correct the faults in the OCR online. When the process is complete, this will make the very detailed content of the material available electronically. Because the words are technical, and because the OCR errors in reading old fonts are not always predictable, once it is complete, there will be an accurate and searchable archive of material which will assist research into biodiversity.
Our task has been to make some prototype tools which would allow the original material and the OCR results sit side by side so that someone can work through and check the taxonomic terms and correct them where needed.
Using our IBM Notes/Domino server and rapid development tools, we have imported the images and OCR text into a database and produced some browsing tools which the Research Group can use to inspect and review the accuracy of the scanning. The programming time from concept from delivery of the prototypes was around 4 days.
(both examples are best viewed using a Webkit browser such as Safari or Google Chrome) CSS and HTML Methods
In more CSS3-capable browsers, the transitions are all smoothly animated, and edges of a sub-page selection are smoothly faded by means of a gradient overlay. In IE, the only non-CSS3-capable browser, the transitions are instantaneous and edges are hard, but everything degrades gracefully. It works fully in any WebKit browser such as Safari or Chrome, and fully without transitions in IE9. Firefox uses a different method to set background image offsets.
Parsing XML input data
The data was supplied in the form of a readily parseable XML file representing all the scanned words and the parallel original images were supplied in a jpg graphics format. We have experimented with both LotusScript and Java methods to import the items into an IBM Notes database. The LotusScript method is faster, although the Java method is likely to have more resilience to changes in the XML schema. The LotusScript method gives startlingly quick results for large volumes. The Java code has been developed separately using IDE tools and then simply loaded into the Notes database to run there. In the database each word is contained in an IBM Notes a document containing the parameters for its location on the page and book. Because it is a readily exploitable database it will be straightforward to alter the stored reading to correspond with the actual word in the graphic after human analysis and selecting from a series of taxonomic alternatives. The CSS and web code behind the working model is dynamic and changes on the fly to give a 'live' feel.
Similar work is being done by the Australian 'TROVE' project
and Google Books
uses some similar techniques to assist in searching for text in antique scanned books.
The programming work has been done by Jef
and testing done by Jacques
The next phase will be to work with the project's researchers
who are analysing the OCR errors heuristically and devising an experimental interface for testers as part of their Community Curation Taxonomy project
Storage and management of back-end data
Here is an image of the database after importing the XML data:
And here is a typical Notes document after parsing. the original XML source line is retained for reference, and fields are reserved for corrected words, and can be added for other analytical purposes as needed later:
IBM's Notes/Domino has a Document-Oriented
data model which can be quickly exposed to the web with only minimal programming. It's one that we have been working with for a long time and it really comes into its own here. it is very similar in concept to the NoSQL
' model developed by Damian Katz, formerly of the IBM Lotus Division.
Notes/Domino is excellent also for working with 'loosely structured' data of this kind. The 'industrial' power and capacity available in IBM Notes/Domino could readily allow it to support many simultaneous sessions and updates as part of the community activity of general 'improvement' in the accuracy of the scanned content.