ICT ITM5 04 0811 - Plan and Monitor the System Pilot.pptx

preventingandeliminatingmuda.pptx

1. Woliso Polytechnic College Information Technology Service Management Level V ICT ITM5 04 0811 - Plan and Monitor the System Pilot Woliso Polytechnic College

2. Learning outcomes to be covered LO1: Prepare for pilot system LO2: Install pilot system LO3: Monitor implementation of pilot system LO4: Evaluate pilot system Woliso Polytechnic College

3. LO1: Prepare for pilot system Establish and Confirm Objectives, success criteria and acceptance criteria for pilot implementation Identify and Secure Technical and organizational resources required for pilot implementation Establish and Confirm Executive support for pilot Complete Project plan for pilot Refer project documentation to appropriate person for approval Woliso Polytechnic College

4. Introduction The effective deployment of ICTs in Businesses and indeed in any setting is a complex affair that goes beyond purchasing hardware and software. Several key elements (see Figure 1) that must be considered if the deployment of ICTs is to have meaningful impact. These components must co-exist; none are optional. Together they conform to a system-wide approach. This approach has to be comprehensive, demand-driven, efficient and well coordinated. Woliso Polytechnic College

5. Woliso Polytechnic College

6. What exactly is a pilot? 'Piloting ' of an ICT project is defined as the implementation of an ICT technology, software, or related project on a small controlled scale to allow for its full impact, benefits and weaknesses to be evaluated before implementation on a regional or nationwide basis. For example, before equipping all Departments or units in a country or region with computer labs, a few school labs can be configured and tested. The testing can measure their performance over a limited period of time. This allows stakeholders and specialists to learn from the experience and refine the requirements for a larger-scale deployment. Alternatively, before deploying a new ICT curricula supported by educational software, the software tools can be installed in a few computers in some classrooms to test how the children and teachers react to it and interact with it . Woliso Polytechnic College

7. Piloting consists in the setting-up of the desired technical environment (hardware, software, content, training, furniture, support material, etc) in a “controlled” space where its performance can be measured. Piloting allows a select group of intended users to interact with the technical environment and find potential problems. What are of significance are the results and the problems that are found in a real world situation, often referred-to as “the field”. Woliso Polytechnic College

8. Why are pilots important? Before investing in a large-scale project, testing its assumptions on a smaller scale can leave us better equipped to plan and execute for the larger scale deployment. We can reduce the risk of propagating mistakes by detecting errors at the pilot stage. For example, we can test if the ratio of 1:3 computers-to-children in a classroom is an appropriate ratio based on student and teacher response. Pilots can be used to assess the impact of the technology on the Businesses or organizations, the people, the community, and whether equipment is used effectively by students and teachers, etc. Woliso Polytechnic College

9. Why Pilot? It is easier to secure funding for a pilot than a large-scale deployment. The pilot can provide the evidence needed to secure more funding or to justify greater expenditure on specific areas such as personnel or baseline studies. The project team members can gain more experience before engaging in a more demanding project. Pilots can be used to compare two or more similar solutions in order to find out which one works best in the field. For example, they can be used to test two similar computer devices or educational software packages at the same time and in similar environments. Woliso Polytechnic College

10. Pilot projects A pilot is a way of testing a theoretical model on a small-scale level, in order to discover potential problems that otherwise would not be detected until full-scale deployment. If these potential problems are not detected on time, it can cost a lot of money and time to introduce changes once the solution has been deployed to more Departments or units. ‘ Pilots ’ are very different from the first stage of a progressive deployment and demand a specific type of planning. Woliso Polytechnic College

11. Steps for successful project plan Every consultant and IS executive has favorite approaches to pilot projects. But the steps for successful pilots, or early phases of reengineering, generally go like this. Begin with a  statement of goals   based on the business purpose that is driving the project. It should include an understanding of how the initial phase--whether called a pilot or not--fits into the total effort. Key stakeholders must be involved now.  Develop an  implementation plan  and solution scenarios, and build  prototypes . Use this phase to create a team with a single-minded purpose and to discover where people still disagree over the business problem or goals of the project.  Woliso Polytechnic College

12. Steps ... Do a  full-fledged design /detail design/ . This involves outline the business rules that drive the process- a tedious but essential step. The design has two components: the user design, making sure you have all the scenarios identified, and the technical design, which should be an iterative process. About halfway through the design phase, the technical people pretty much take over and bang out details, but they should be always checking back with the end users and business-side owners.  Devise a  project plan and budget . Get approval of them before going further. Woliso Polytechnic College

13. Steps .... Next comes the  construction phase . During this stage it's important to have weekly meetings of the stakeholders to monitor status and make decisions. Begin communicating with, demonstrating for and training the broader user community before the system is completed.  Participate user testing . During this phase, the end users get deeply involved again. Let them bang away on the system and determine whether it does what they said it should do. This stage is over when the system stabilizes from the usability standpoint and when the customer is comfortable that it meets business needs.  Woliso Polytechnic College

14. Steps .... Handover the project - the pilot is over when the system is in the users' hands and the  business transformation  begins.  Conduct a  post mortem , talking to users, business managers, developers and development managers. Evaluate the lessons learned to improve procedures as you move forward. Woliso Polytechnic College

15. Pilot Project Plan A pilot project requires the same degree of planning as any ICT project in order to ensure success. As with any other ICT project, the pilot’s objectives Scope tools implementation details Logistics funding sources ; installation ; support ; maintenance ; replacement plan ; team members profiles and responsibilities ; budget and timeline all have to be documented. Woliso Polytechnic College

16. In the Planning stage: Objectives - the initial supporting theories to be tested, for example, that the software improves the learning process or that less children leave school. This is of course associated with the educational objectives that the ICT project has to support. Timeframe - Define the timeframe for the pilot. The period should be of significant duration for results to be produced and measured. Normally a pilot will run for one or two school terms. Selected sample Units or departments: A sample size of department must be selected in accordance with the type and purpose of the pilot project and the initial theories that are to be. It should be a representative sample, that is, a sample that reflects a realistic representation of the variations within the relevant organizational units Woliso Polytechnic College

17. Pilot project planning stage ... Team : determine the team members that will execute the pilot, and the roles and responsibilities of each of them Budget : estimate and allocate sufficient funds for the pilot to be executed effectively. A pilot can turn out to be more expensive per department or test site than a large scale deployment would finally be, since a large amount of money has to be invested in planning and monitoring. A 15%-20% of the total amount has to be allocated for extras that are normally very difficult to estimate at the beginning of the project. Woliso Polytechnic College

18. Pilot project planning stage ... Monitoring and Evaluation plan: including the indicators that are going to be used to measure the results of the pilot. This is usually quite difficult to define, since we will not only have to find the indicators but also determine the way to collect and process the information as accurately as possible. Woliso Polytechnic College

19. LO2: Install pilot system Once the pilot planning document is ready, the resources are in place, and some candidate business units have been selected, it is time to actually run the pilot. The execution will usually include these stages: Pre-analysis: final selection of the sample Departments or units where the pilot will run. Analysis of the business’s infrastructure that is in place and execution of required adaptations if necessary (i.e. buildings, office infrastructure, specific furniture, electricity provision). Set-up: acquisition, transportation, installation and configuration of the equipment and/or software. Woliso Polytechnic Colle ge

20. Project presentation: several informative talks with people directly and indirectly involved like teachers, headmasters, parents, community members and students about the objectives of the pilot, how long it will last and how to record the experience, etc. It is particularly important to explain carefully the objectives and how crucial it is for those involved to record the failures and problems they might experience, as well as the successes . It is also important to clarify the time-frame and the implications for the end of the pilot as well as the responsibilities of the stakeholders. Woliso Polytechnic College

21. User training: Training users to use the specific tools. Normally the teachers are trained first, and they in turn, train the students if needed. It is best if teachers are provided with training materials also. Execution: Once the pilot is running, the monitoring plan is activated and the indicators have to be measured and data recollected and analyzed. The team will have to provide support for teachers, students, parents and the community. Early adjustments can be required to the project Plan if some major issues turn out that require it. Woliso Polytechnic College

22. End of pilot: by the end of the determined period, if the data gathered is considered sufficient for analysis then the pilot is terminated. At this point the equipment is removed, with due regard for careful explanation to teachers, students and the community as to the rationale behind this. Woliso Polytechnic College

23. LO3: Monitor implementation of pilot system One of the key tasks that has to be planed for is the development of the Monitoring and Evaluation Plan, or Monitoring and evaluation plan Monitoring and evaluation is the way of measuring the outcomes of the pilot in order to match them to the intended goals. This Plan should define the data to be collected and how is it going to be collected. The variables that will be analyzed are called “indicators”. Some of the indicators will have numeric values (quantitative) and will be easier to measure (i.e. number of computer hours per student in a week, numbers of equipment that failed in a given period), while other qualitative indicators will be much harder to measure (i.e. Customer satisfaction with the tool, or how the service process or product quality was improved by the use of the new software). Woliso Polytechnic College

24. How to collect and measure data As we mentioned before it is critical to design a proper Monitoring and Evaluation Plan and use the pilot for the collection of relevant data that will allow for the verification and validation of the original ideas proposed by the project. Below various collection methods are outlined: Automatic : it is possible to have automatic tools that will collect some numeric data like number of hours a system was used, exactly when and by whom (if each student is given a username for example). This way data does not have to be collected from users directly. External: One or several observers from the team can be present during teaching hours or at the students’ homes. This can be done with or without interfering with the normal teaching process. Woliso Polytechnic College

25. Subjective : Participants can document their impressions, experiences and ideas through several tools like journals or diaries, questionnaires, tables or forms. The frequency of documenting must be pre-defined. This type of feedback gathering is useful for quality metrics. In addition to free-text comments that could enrich the conclusions, this type of feedback can be tabulated and analyzed as well. Environmental: Data can also be gathered by questioning teachers, students, headmasters and also the parents and the community about the pilot results and their perceptions. Woliso Polytechnic College

26. LO4: Evaluate pilot system While the pilot is running and being evaluated, there are other things the team can be doing as well, in order to provide additional information about the products to be tested. Technical testing: If new hardware is involved, perform technical testing of the equipment: install a lab or hire another organization to test the equipment for errors and problems. Try to find the limits of the electronics by doing a stress test (can be destructive in some cases). Test for extreme environment situations (dust, heat, water) and rough handling. Test the electric consumption and battery life. Try to estimate the useful lifespan of the devices and that of its component parts. Try to work on tentative repair and replacement procedures and related costs. Woliso Polytechnic College

27. LO4 - Contents Woliso Polytechnic College 4.1 Review Pilot objectives and success criteria against pilot operation 4.2 Review Client and executive feedback on pilot against acceptance criteria 4.3 Identify and prioritize areas of success and improvement 4.4 Assess the impact of changes on the pilot 4.5 Document review process and submit to appropriate person

28. Software testing : If software is involved, test for hardware requirements and optimal hardware setup, installation/reinstallation processes, possible errors and bugs, execution under different hardware and operating systems, weaknesses in the design and security, and language translation mistakes or needed adaptations. Work on initial versions of training manuals, FAQ and online help. Universities and online user communities can be invited to help with these tasks. Research: perform an online research on similar experiences with this product in other countries. Ask the hardware/software provider for contact data from other customers and contact them. Try to extract what can be learned from previous experiences. Woliso Polytechnic College

29. Nationwide costs estimate: Undertake a tentative analysis of the estimated logistics and costs for nationwide deployment, taking into consideration what is being learned from the pilot. This analysis should include the initial investment as well as running costs like electricity, continuous training, software licensing and support and maintenance. A variation of +/- 50% in the amounts estimated nationwide is normal at this stage. Woliso Polytechnic College

30. How to ensure the success of a pilot project “Success” of a pilot can be defined as the smooth running of the pilot as planned in an adequately selected sample of Departments or units/students, producing results that can be trusted for the variables/indicators selected in the Monitoring Plan. Some of the things that can be done to ensure that the pilot runs smoothly and yields valid results are: The monitoring plan is in place and the indicators to be measured have to be properly defined, and the pilot team has to be thorough and honest in collecting required data to measure the indicators. Have observers from the team in place to verify the collected data. Woliso Polytechnic College

31. Do not disregard the importance of the human factor: one common mistake is to unconsciously convey to participants that the only results desired are positive ones. This might result in participants hiding problems, producing might hide the problems that they find and this can produce an artificially “successful” pilots with disastrous large-scale consequences! Explain carefully to all participants the objectives of the pilot. Talk to employees, customers, stakeholders and the community. Describe the steps to be taken when they find errors or problems. Make it clear that it is not their fault, and make sure they have the tools to report both positive and negative results. Woliso Polytechnic College

32. Do not rush: give pilots the time that they deserve. Normally relevant results cannot be produced in three weeks (such as the use of software). Give pilots enough time (one or two school terms or semesters) to find significant results and detect potential problems. Involve the community : of course it depends on the type of solution, but it is usually critical that the community gets involved in the pilots as well. In this way, the support of parents and stakeholders can be secured. Woliso Polytechnic College

33. Summary Outline of the structure of a Pilot Planning document Projects Name Introduction Project History : how this Pilot originated Project’s objective/s: Normally business or organizational objectives and mention of how the project specific objectives will support them. Scope: number of institutions, location, classes involved, number of teachers and students involved. Planning Detailed description of the IT solution characteristics Detailed deployment plan Team members profiles, roles and responsibilities Budget and funding sources Timeline (preferably using some planning technique like GANTT) Execution Installation procedure and standards Support and maintenance, replacement plan Implementation details Training Plan Logistics Monitoring and Evaluation Woliso Polytechnic College

34. The End of the 4th UC Woliso Polytechnic College