You may be feeling defensive and uncomfortable at the prospect of your project being evaluated.

It feels as if you’re heading to court where you’ll be cross-examined, criticised and then judged. Perhaps you, or your project, will be found wanting with uncomfortable consequences for everyone.

Perhaps you’ve tried to push this spectre out of the way, meanwhile your doubts grow larger each time you consider it. Now you can’t put it off any longer.

The good news is there’s no to need to fear an evaluation. This is because a professional evaluation involves a series of steps before coming to any element of judgement. These steps are:

• understanding the context of your project – why it was set up and how it operates
• agreeing with you a shortlist of important project objectives to review
• reviewing existing information on your project, from your project records
• identifying what extra information is needed and how it will be obtained, usually information and opinions of: clients, project staff and stakeholders
• discussing and agreeing survey plans with you
• impartial analysis of all the information that’s collected
• drawing conclusions and making recommendations based on all the preceding steps.

Professional evaluations also follow good practice guidelines, like those of the UK Evaluation Society, which protect the interests of all parties.

So, while evaluation does eventually involve an element of judgement, like a good court case it is based on first collecting sound evidence. Unlike a court case, there is no fierce cross-examination, and no desire to reach the black and white extremes of a ‘Guilty’ or ‘Not guilty’ verdict.

The bottom line: Please be assured that evaluation is a fair process, involving full discussion with a range of parties, with conclusions emerging logically from the information and opinions collected.

© May Johnstone, 2008, mayjohnstone.co.uk. Please feel free to circulate this article provided it is used in its entirety, including this acknowledgement.

Imagine you had commissioned builders to build your house extension; would you just leave them to it? More likely you would want to check how things are going, maybe even adjust things a little as you go along.
You need to keep tabs on the research project, so you know it’s going along the right lines, keeping to time and budget, and to respond to queries and problems as they arise.

You may be able to share this overseeing responsibility with someone else in your organisation, or on your project board. Here are five things that need to be done.

1.    Be available

Particularly at the start, your researchers will need access to you or someone in your organisation, to provide background information, and essentials like client lists, details of programmes you run, provide feedback on their draft questionnaire, etc.

Later your involvement is likely to diminish, though there should always be someone available to respond.

2.    Make your contributions

However much you want the researchers to take the work off your hands, there are bound to be tasks which have to be done by someone in your organisation. If the researchers are to keep to their timescale, they depend on you to provided lists, and give feedback on their plans and drafts quickly. Delays at your end will introduce delays in their work.

3.    Formal review

Unless you have commissioned a very small piece of research, you have probably built in some formal checkpoints. These are likely to be milestone events such as completion of a stakeholder workshop, or completion of key chunks of work such as gathering data.

You may have scheduled meetings to review progress, perhaps requesting a verbal or written progress report.

These milestones are often tied into payment schedules, making them straightforward to monitor on both sides.

4.    Informal contact

If you set up a good relationship with your researchers, they will feel encouraged to keep in touch with you. This is time well spent, since it means that if things do go off-course you’ll hear sooner rather than later.

If you are unavailable for more than a few days it’s a good idea to give your researchers another contact.

5.    Troubleshoot

If you or the researchers feel there’s a challenge to be tacked with the research that is beyond their remit, you will need to make time to have an input, and perhaps make a decision.

The bottom line: Being available, ensuring your organisation makes its contribution, keeping formal and informal checks on progress, and troubleshooting will ensure your research runs smoothly, and that you don’t get any nasty surprises at the end.

A good survey response rate can be tricky to achieve.

You know your respondents are busy people, and you’ve heard that a proliferation of surveys has led to ‘survey fatigue’.

Yet your clients’ feedback is critical to the continued funding of your project. 

Remember the last time you received an email asking you to ‘complete a quick survey’, or you opened an envelope to find a questionnaire inside, or you answered the phone to hear the dreaded “Please could you spare a few minutes…”.

How did you feel?

The trick to getting a good survey response is to put yourself in the shoes of the recipient. Then you can identify what would make you (ie your respondents) more motivated to participate.

Three tried and tested strategies to increase your response rate

1. Provide an incentive

People are busy, they need a good reason to give up time to answer your questions. Providing an incentive is a good ‘carrot’ to attract people who may be swithering whether or not to take part in your survey.

Incentives can take many forms: from money to vouchers to goods to a free service or subscription

• Check your organisation/funders policy on incentives – if it excludes spending money on them, either find something you can offer ‘in kind’, or get an organisation or business to donate the incentive.

• Consider the target group for your survey, and brainstorm a list of things they would value. I have used gift vouchers, money, books, and a free session.

• Describe your incentive in an attractive way, and highlight it early on in the Introduction to your survey.

2. Make it easy

The easier your questionnaire is to complete, the more people will fill it in. Any barriers provide an excuse to ‘put it on one side’. We heard of a researcher who received a questionnaire 6 years after the close of the survey!

• Start with a question which is very easy to respond to.

• Use ‘tick box’ responses for the majority of questions.

• For postal surveys, provide a ‘post paid’ return envelope.

3. Keep it short

The shorter your survey, the more responses you are likely to get. This is particularly the case with a printed questionnaire, where the respondent can immediately see how ‘long’ it is.

• Test your questionnaire with a few ‘real’ respondents to find out how long it takes to complete. If it’s more than 10 minutes consider pruning your questions (though an acceptable time will vary for different audiences).

• Include the time the questionnaire will take to complete in the Introduction to the survey.

• Be creative with layout to make the questionnaire look ‘short’.

The bottom line: By providing an incentive, and making your questionnaire quick and easy to complete, you will increase the number of people who respond. This in turn makes the feedback from your survey more robust.

Once you’ve chosen your survey method, and developed your list of questions, it’s tempting to get right on with the survey.

However well you and others have thought out your survey, there’s bound to be at least one glitch to resolve. Much better to take the time to identify them at the beginning than find them after you have run your full survey.

Whatever your survey method – phone interviews, face to face interviews, printed questionnaire or online survey – here are seven pitfalls you’ll want to avoid.

1.    The wording of a question doesn’t work

Interviewers will quickly tell you if the wording of a questionnaire isn’t working, because respondents will have asked for clarification. For self-completion surveys, you can tell you need to improve the wording if you get unexpected responses to a question, or perhaps no responses at all.

For example, it may become apparent the age bands: 18-25, 25-30, 30-40, over forty overlap, and should be 18-25, 26-30, 31-40, over 40.

Other problems could be: ambiguous questions, unclear questions, use of abbreviations respondents don’t understand, etc.

2.    A question is out of sequence

Some questions build on each other, or follow a logical sequence.

For example, if Q10 asks ‘Which of the following benefits did you receive from the scheme?’ it should come after Q12 which asks ‘Which scheme did you participate in?’

3.    The routeing doesn’t work

If your questions involve routeing, for example, ‘If No, please go to Q10’, there is potential for a mistake, especially if the questionnaire has been through a number of drafts with changes in the numbering of questions.

Online surveys have the scope for more sophisticated routeing (called ‘logic’ in Survey Monkey), and consequently more scope for this type of error.

4.    Questions overlap

If you get responses like, ‘see Q3 above’, you probably need to rationalise overlapping questions.

For example, if you ask ‘What suggestions do you have to improve the scheme?’, and later ask ‘Do you have any further comments?’ when it comes to analysis, suggestions will appear under both questions. The solution might be to amend the second question to ‘Do you have any further comments, not answered by previous questions?’, or if there were few responses, to delete the second question altogether.

5.    Generating hundreds of ‘Other’ responses to categorise

If you devise questions with multiple choice responses (which are good because they are quick to analyse), it’s often necessary to include an ‘Other’ box at the end. A pilot is a good opportunity to review the ‘Other’ responses, and extend the list of multiple choice responses. This saves a lot of time categorising responses later on.

6.    Your survey data can’t easily be analysed

As well as collecting data, a pilot should also analyse the data, to check the analysed data makes sense.  For example, analysis of your pilot data may show there are very few responses to one of the choices in a multiple choice list, so it should be excluded.

7.     There’s a glitch in the logistics of your survey method

A university piloted its online students survey, and wondered why only males had responded. It turned out the email list of students had a filter by sex, and this had been ‘turned on’ when the pilot list was sent out!

However straightforward your survey method may seem, it’s only by testing that you can be sure it works. In my experience the glitches are usually small things, but they could have significant consequences.

The bottom line: Piloting your survey  will help you resolve any glitches, thus saving time in your main survey, ensuring your results are robust, and perhaps avoiding an error which would invalidate your survey results.

Chances are the upcoming evaluation of your project is going to put a strain on your time and resources.

You want it to go smoothly, so you can keep your other work moving. Here are five steps which will help you keep your evaluation hassle-free. 

1. Get people on board

Getting people on board at the outset saves hassles later on. You don’t want to work all the way through the evaluation, come up with a good set of recommendations and then find senior management have other priorities, or project staff are resistant to the proposed changes.

Keys

• Prepare your reasons why the evaluation will benefit: your organisation / the project / partners / stakeholders / other parties, taking into account the wider picture of these parties, and how your project fits in.
• Be prepared to genuinely listen to difficulties the various parties may raise, and then seek ‘win-win’ solutions to these difficulties. This may involve modifying your plans.
• Get people on board at the outset, that way you can create a positive environment from the start.

2. Foster a positive environment

A positive environment around the evaluation will help diminish the threat that some people feel when they hear the ‘evaluation’ word. Those closely involved with the inception of the project, and those running it on a day to day basis may think of it as ‘their baby’, and feel defensive at the thought of it being ‘assessed’, particularly if an outsider is involved. This might discourage them from engaging fully in the evaluation.

Keys:

• Tell people about the evaluation as early as possible, and take every opportunity to remind them about it.
• Keep them fully informed as the evaluation progresses, particularly of changes.
• Show your own positive attitude in practical ways, and make it clear you’re personally learning from the evaluation.

3. Treat the evaluation as a learning opportunity

A good way to take the threat out of the evaluation is to treat it as a learning opportunity. In this way ‘criticisms’ can be turned into ‘suggestions’, and ‘mistakes’ into ‘lessons for the future’.

Keys:

• Proactively adopt a no-blame culture yourself; where mistakes are OK so long as lessons are learned.
• Encourage this learning culture within your team and wider sphere of influence.
• Include an aim ‘to learn from the evaluation process’ in the evaluation specification.

4. Allocate time to the evaluation

If you’re short of time you may plan to contract some or most of the evaluation out to a consultant. If this is the case, you’ll still need to spend time helping the consultant understand what your project aims to achieve, what your stakeholders are looking for etc. Otherwise you may end up with an evaluation report which isn’t particularly helpful to you.

Keys:

• Schedule enough time to ensure any consultant involved really understands your project – how it originated, its aspirations, and how it works on a day to day basis, They will also need to know how your organisations works.
• Schedule admin time to collate lists of clients, stakeholders, project achievements, and any monitoring data. It’s very unlikely an outsider can do this for you.
• Schedule regular times into your diary to be in touch with the consultant. If everything runs smoothly this time may not all be needed; if there are hitches you’ll have the time to deal with them.

5. Develop a specification for your evaluation

Without a spec, like a meeting without an agenda, evaluation feedback will drift off in all directions, and you’ll have a hard job drawing any conclusions.Even if you plan to carry out the evaluation yourselves, you’ll need a specification of what it aims to achieve, as well as any constraints such as timescale and budget.

You can use the evaluation specification as the basis of a contract with any consultant you decide to appoint to help with the evaluation.

Keys:

• Prepare the evaluation aims and objectives in consultation with colleagues, project staff, funders and any partners. This will help get them on board, and the whole process will focus attention on your project.
• Focus the evaluation on key project objectives; usually it’s impractical to check out achievement of all of a project’s aims and objectives.
• Include timescale and budget, and any suggestions you have for the evaluation methods. Be open to revising the specification if needed as the project progresses.

You’ve probably noticed that three of the five steps to a hassle-free evaluation are ‘people’ factors: getting them on board, fostering a positive environment and treating the process as a learning opportunity.

Taking steps to address these ‘people’ aspects greatly increases your chances of setting up the evaluation, running it and achieving positive follow-up. Someone inside the organisation needs to be involved in addressing these steps; they can’t readily be achieved by an outsider, though an outside perspective can certainly help.

Of course there can be all sorts of technical hitches too, and an experienced evaluator will help you take care of these. Allocating time to the evaluation and having a clear evaluation specification will minimise these hitches.

The bottom line: Address these five steps and you’ll be well on the way to a hassle-free evaluation!

You have six tenders sitting in front of you

This task reminds me of going to buy a pair of jeans for my teenage daughter. First I need to be very clear of her size and current fashion preferences, then there’s a fair chance I’ll get a pair she likes!

Here are my four tips for checking out tenders

1.    Be open to different approaches

If you already have a picture of the way you feel the research should go, put it aside while you check out the approaches your respondents have come up with.

It’s quite possible there’s another approach you hadn’t thought of, which would save time or money, or produce results that you hadn’t thought it would be possible to get.

For example, a respondent suggests piggybacking a workshop onto the project’s Board meeting, giving an ideal opportunity to invite the other key stakeholders, and hold a mind workshop to explore their views.

2.    Set up a logical selection system

Hopefully, you gave respondents the criteria you would use to assess proposals. Just like interviewing candidates for a job, it’s good to use a points system, so your subjective impressions don’t take over.

For example you might choose to allocate points as follows for each of the following criteria,

30%    Appropriate method
40%    Relevant experience
20%    Cost
10%    Available start date

If several people are reviewing tenders, each one applies the criteria, so you have an objective assessment from each person.

3.    Interview, if necessary

You may want to leave the decision whether or not to interview until the written proposals have been assessed (though it’s prudent to have given respondents the date to pencil in their diaries in case).

If there is a clear front runner it may not be necessary to interview.

If you do decide to interview, you may want to use the same criteria again, with the ability to change scores given as you question respondents.

You may also want to add in one or two further criteria, such as:

• assessment of relevant interpersonal skills (eg listening, questioning, engaging)
• expansion of relevant experience given in the written tender
• flexibility to accommodate different viewpoints

4.    Inform everyone of the outcome

Aim to contract with your preferred respondent quickly, so you can then let the other respondents know they were unsuccessful

It’s good practice to give unsuccessful candidates feedback on the strengths and weaknesses of their tender. This provides some recompense for the time they invested preparing their tenders.

The bottom line: You can increase the chances of selecting the most appropriate tender by: being open to new approaches, having a logical assessment system, and interviewing if necessary.

It’s pretty much the same with research, you need to know where you are now, what you want to know, and what you’ll do with the information you get.

1.    How does your project fit into the big picture?

If you’re a hands-on project manager, it’s easy to get so engrossed in the detail that you forget to stand back from time to time and check how it fits into the bigger picture.

Whether you want to research the feasibility of a new project, evaluate the achievements of a two year project, or something in between, it’s essential to step back and check how your project fits into the bigger picture of your organisation, and any project partners, and any other key stakeholders.

Funding and co-operation from within your organisation, and perhaps from partners, depend on your project contributing to their aims or policies. It’s a good idea to update yourself on their current policies/aims so you keep your project research aligned with this bigger picture.

Knowing where your stakeholders are heading, will influence the questions you want the research to tackle.

2.    What questions do you want answered?

Next you need to get a clear picture of the questions you want the research to answer. Broad questions like: ‘Would it be feasible for us to move into our own premises?’, or ‘Has the training pilot been a success?’ are often the start point that initiates the research.

However, they need to be broken down into a series of more specific questions. For the premises idea, these might be:

• What are the advantages and disadvantages of buying, renting and leasing premises?
• What are the capital and running costs of each option?
• Would there be scope to earn revenue?
• Overall, would it be feasible for us to have our own premises?

For the training pilot:

• To what extent did the training pilot meet its original objectives?
• What improvements were made to the programme during its two years of operation?
• How could it be developed for the future?

3.    What will you do with the answers?

When you initiate research it’s easy to get waylaid by the statement ‘Wouldn’t it be nice to know…’ Committee members, who will probably comment on your research proposal, often come in with suggestions from their varied agendas.

If you want to avoid your research growing from a molehill into a mountain, you need to be very clear how the information will be used. That clarity will help keep the research focused.

Broad uses of project research include:

• justifying stakeholder funding and support (feasibility studies)
• getting a profile of project clients (baseline studies)
• learning from the project so it can be improved in future (evaluations)
• learning from a pilot project, to develop future policy
• reporting back to funders (evaluations)

The bottom line: Getting clear on the big picture, the questions you want answered and how your research will be used provides a firm foundation for moving forward  to commission research, or develop your own research plan, or obtain funding for your research.

It’s easy to put off the task of writing up that report!

Here are six tried and tested tips to make report writing more approachable.

They will help you get started quickly.

1.    Decide on your structure

Sorting out the structure first means you’ll know where everything has to go. It’s rather like planning your journey in advance, so you don’t end up taking detours or ending up in dead ends. Here’s my suggestion for a structure, which you can modify to suit:

1. Title page
2. Contents
3. Introduction
4. Executive summary
5. Methods
6. Results
7. Conclusions & recommendations
8. Appendices

2.    But don’t write in that order!

It would be difficult and daunting to write a report from beginning to end. I suggest you set up the headings for the different sections of your report, and then get started on a section which feels easy to you.

• You might start with:

The Title page and Contents page.
Introduction, Methods, and any Appendices relating to the Introduction or Method

• Then move on to:

Results, any Appendices for Results.

If you have a lot of detailed results, which warrant an Appendix, you’ll probably need to write up the Appendix first, so you can summarise it into your Results section.

• And write these parts last:

Conclusions and recommendations, Executive Summary

3.    Structure each section before you start

Just as you have structured the whole report, it will help you to write the main headings within each section. For example a Results section might include:

• Trends from desk research
• Employers’ views
• Trainees’ views
• Stakeholder views’
• Summary of results

4.     Ask for help

If you find it difficult to structure your report, ask a colleague to help. You don’t necessarily need someone familiar with your project – explaining things from scratch will give you insight into a reader’s perspective.

5.    Start writing anywhere within a secton

It can be hard to write the first paragraph of a section, partly because it’s an overview of a section which you haven’t written yet! Word processing makes it easy to start anywhere. I usually start with a part I can write easily. Then I add bits in, moving them around as necessary.

6.    Use technology

Computer software offers many ways to make writing easier. Here are my favourites:

• Keep a backup of your report
• When you have added substantially to your draft, save your report as another version, so you end up with versions 1.1, 1.2, 1.3 etc. This means you can easily go back to earlier drafts if necessary.
• Use styles for different levels headings, so they are consistent.
• Use Word’s facility to automatically generate, and update, your Contents page (Insert/Reference/Index and Tables).
• Use Word’s facility for Footnotes, to add short pieces of information which interrupts the flow of the text, at the bottom of the page (Insert / reference / Footnote)
• Use Word’s ‘Track changes’ facility if you want to circulate your draft and be able to see each other’s changes (Tools / Track  Changes)
• Use Word’s Help menu to learn to use any of the above features.

The bottom line: Report writing doesn’t need to be daunting. Once you have a structure, the trick is to start with a part of the report which feels easy to you, and build out from there. A range of software tools will help you.

You’ve collected and analysed the research information your project needs, now the prospect of writing it up feels a bit daunting.

The good news is there’s no need to reinvent the wheel.

Here’s a tried and tested template that works!

Contents for a project research report

1. Title page

This should include:

• title of the research
• date
• who commissioned the research
• author(s)

2. Contents

This is a list of all the sections of your report. You may wish to include sub-sections for a long report.

3. Introduction

The aim of your Introduction is to introduce your project, and the research, to someone who knows nothing about your project. It is likely to include:

• background to your project: its aims, partners, activities to date etc
• broad reason(s) for the research
• the research brief

4. Executive Summary

The Executive Summary is a very concise summary of your whole report, usually between 1-4 sides, depending on the extent of your research. It is usually the last section of your report to be written.

Some people will only ever read this part of your report; it may be circulated as a separate document. It therefore needs to include a brief introduction setting the context, and a very brief description of research aims and methods, and then the key results and conclusions. Recommendations are usually given in full, since they are often the main focus of interest for readers.

5. Methods

In this section you describe the methods you used to collect data, including desk research, surveys and consultations. If relevant, include quotas set, samples achieved, number of people consulted. If there is extensive detail, consider separating it into an Appendix.

6. Results

For a small research project, you include a summary of all your results here. For a larger project, you would separate the full results into an Appendix, with a summary, or series of summaries, in this section.

Where possible summarise your results as lists or tables, perhaps enriching with some quotes from respondents.

7. Conclusions and recommendations

Your conclusions summarise the main points emerging from your results, in relation to the research brief.

Your recommendations specify the recommended actions, which emerge from your research, justifying why they are being made. They are usually numbered, and made to stand out using bold or a different font.

8. Appendices

Appendices are used to ‘house’ detailed information, which only some people will need to read. This often includes: detailed methodology, and detailed results.

The rest of the report should make sense to a reader without reading the Appendices. The main part of your report may well end up being circulated without them.

But don’t write it in order!

While the above list defines your content, the order in which you write the sections of your report will probably be quite different – it’s often easiest to start in the middle!

House style

Some organisations have a house style for reports, if that’s the case, check it out as it may require some variations to the structure suggested here.

The bottom line: Outline the structure of your report, before you start writing, and it will be straightforward to slot everything in the right to place, to produce a digestible report.

]]> http://mayjohnstone.co.uk/a-foolproof-structure-for-your-research-report/feed/ 0 http://mayjohnstone.co.uk/checklist-for-preparing-a-project-research-invitation-to-tender/ http://mayjohnstone.co.uk/checklist-for-preparing-a-project-research-invitation-to-tender/#comments Wed, 15 Oct 2008 15:18:49 +0000 admin http://mayjohnstone.co.uk/?p=95 Summarising what your project is about, and asking questions of potential evaluators can feel daunting.

This checklist will help you benefit from the experience of others, put your invitation to tender together quickly, and receive robust tenders.

Check out your organisation’s procedures

If your organisation has a Procurement Officer, or uses online tendering, check this out first. There will probably be a set of procedures you need to follow, and perhaps a template, to make it easy for you.

And it’s still a good idea to check the rest of this list, since it’s possible I’ve included some ‘extras’ you’ll find helpful.

Put yourself in your reader’s shoes

You need to assume the people who will read your invitation know nothing about your project. This probably means spelling out things that are obvious to you, but very necessary for your readers. The better your invitation, the fewer people you’ll have ringing you up or emailing you questions!

Give the background

A concise background helps readers know how your project developed, how it fits into the bigger picture, and why the proposed research has come about.

It’s a good idea to include links to relevant websites and reports.

State the aims and objectives of the research

Include a clear statement of the general aim(s) of the research, and more specific objectives. Be as specific as you can.

For a complex project refining the aims and objectives sometimes forms part of the research.

Include any guidance on methods

If you have already worked out methods for the research include them as suggestions, and allow respondents to offer alternatives. There’s always a chance they may have an even better idea!

Don’t worry if you don’t have ideas –your respondents will certainly make suggestions based on their experience.

Ask for relevant skills and experience

You will feel more confident if you appoint a person, or team of researchers, with expertise of similar research, preferably in your sector.

Include your timescale

Check on the events which will determine the completion date for the research. These might include: a funding deadline, a Board meeting, the start of a new term, a workshop where recommendations can be debated, etc.

You also need to be realistic, for example: it takes time to pilot and conduct a survey, or to get people together for a workshop, etc.

Include dates for the milestones you expect the researchers to achieve along the way. For example: methodology piloted, interim report submitted, etc.

Include your budget

It’s best to give an idea of your budget limit, so the tenders you get all lie within it. Otherwise you may get wildly disparate estimates for the research.

Give a realistic timescale for responses

Allow at least three weeks for responses, or you may miss out on replies from good consultants who are away, or busy.

Explain how you will assess tenders

Common criteria include:
o    Familiarity with your sector/subject area
o    appropriateness of methodology
o    expertise of tender team
o    budget
o    risk assessment

If you propose to hold interviews for a shortlist of respondents, include the date with your invitation, so respondents can keep it free.

Be available for discussion

Unless your organisation precludes this, make yourself available to discuss the research with respondents. This gives them the opportunity to provide you with a fully developed tender, and avoid misunderstandings which can occur where dialogue is not allowed.

Get someone to read your tender over

This could be a co-worker or someone from your Board or Committee.

The bottom line: Following the above twelve points will help you to attract tenders containing the information that you need to move ahead with confidence to commission your research.