Vendor Evaluation Worksheet

How to use this worksheet

Vendor demos are optimized to be impressive; this worksheet is designed to make them comparable. It gives you fifteen yes/no capability checks, a fixed set of questions for demos and RFP responses, and a blank scorecard. Fill in one copy per vendor and every candidate ends up judged on the same evidence, not the same pitch.

Work it in order: run the capability checks against each vendor's public documentation first, take the demo and RFP questions into your calls, then score. Print one copy per vendor, or copy the sections into a document and duplicate the page. Write down what you were shown, not what you were told — if a claim cannot be demonstrated or put in writing, treat the answer as no.

The worksheet is vendor-neutral. It applies to any app push provider, and just as well to adjacent channels — if you evaluate a wallet-pass channel such as PushNotice, hold it to the same fifteen checks.

The fifteen-point capability check

Answer yes or no for each vendor before the demo, using their public docs and pricing page. Anything you cannot verify in writing goes on your demo question list.

• Supports every platform you actually ship

  iOS via APNs, Android via FCM, and web push if you need it — available on the tier you would buy, not only the enterprise plan.

• SDK is current and maintained

  Check the changelog date and release cadence. An SDK that has not shipped a release in many months is a risk every iOS update.

• Reports delivered, not just sent

  Sent counts include dead tokens. You need delivered and opened as separate numbers on every campaign report.

• Segments on your events, not just device data

  Targeting by OS or country is table stakes. You need segments built from your own custom events and profile fields.

• Frequency capping and quiet hours

  Per-user send caps and local-time quiet hours that you configure — not a global default you cannot change.

• Granular opt-out categories

  Users can mute one message type, such as promotions, without losing another, such as order updates.

• A/B testing with a holdout

  Variant testing plus a no-send control group, so you can measure lift rather than just compare open rates.

• Real-time event triggers

  Automations fire within seconds of the triggering event, not on the next scheduled batch run.

• Full API access on your tier

  The complete send-and-query API with published rate limits, included in the plan you are actually quoted.

• Raw data export

  Event-level export via webhook, warehouse sync, or API — without being forced onto an enterprise contract.

• Dead-token hygiene

  Uninstalls and expired tokens are detected automatically and stop counting toward your bill and your delivery stats.

• Compliance documented

  GDPR and CCPA posture in writing, a signable data processing agreement, and clarity on where your data is stored.

• A billing metric you can predict

  Know whether you pay per MAU, subscriber, device, or send — and model a strong growth month on each before comparing prices.

• Free-tier limits in writing

  Exact caps on sends, subscribers, features, and data retention — and whether notifications carry vendor branding.

• A documented exit

  You can export device tokens and subscriber data in a standard format if you leave. Ask before you sign, not after.

Demo questions: watch, do not listen

Send this list to every vendor before the call and ask the questions in the same order. Each answer should be something you watch happen on screen. Record what you saw next to each question.

RFP and pricing questions

These belong in writing — in the RFP response or the order form, not in a call summary. A vendor who will not commit an answer to paper has answered.

The scorecard

Score each vendor 0 to 5 per criterion, using only the evidence you recorded above. Write the score in the vendor columns; sum each column at the bottom. If some criteria matter more to you, double their score before summing.

Run the evaluation

The worksheet only works if every vendor gets the same process. This is the sequence that keeps it fair.

1. 1

   Shortlist three vendors

   More than three and the demos blur together; fewer and you have no negotiating leverage. Use the fifteen-point check to cut the long list before booking calls.

2. 2

   Send the same agenda to every demo

   Share the demo questions in advance and hold each vendor to the same script in the same order. Deviations tell you something too — note what they steered around.

3. 3

   Score within 24 hours

   Fill in the scorecard the same day, from your written notes, before impressions fade and the pitches merge in memory.

4. 4

   Trial the free tier before you negotiate

   A week on the real product with real data answers questions no demo can — especially where the free-tier caps actually bind and what upgrading is like.

5. 5

   Check one reference off-list

   Ask each finalist for a customer at your scale, then also find one the vendor did not supply. Ask both the same question: what surprised you after signing?