CryptoCard service status

Visitors on certain networks could not reach the site at all, while others saw no problem. Balances and cards were never affected.

Website Dashboard API v1

1. 18:38
   Resolved Routing is restored and the site is reachable everywhere we can test from.
2. 18:34
   Identified Our edge provider is having a routing issue in one region. We have opened a ticket with them and are failing traffic over.
3. 18:29
   Investigating We are getting reports that the site is unreachable from some networks. We can still reach it from ours, which points at the edge rather than at us.

Card production was paused for part of the day. Orders placed during the pause were queued and dispatched afterwards, in order.

Physical card fulfilment

1. 16:15
   Resolved Dispatch has resumed and the queue has been worked through. Delivery estimates shift by about one working day for the affected orders.
2. 14:04
   Identified The pause is on their side, on a batch printer. No order has been lost and nothing was charged twice.
3. 12:37
   Investigating Our card producer has paused dispatch this morning. Orders are still being accepted and are queued.

Confirmation codes for online payments arrived several minutes late. Codes issued during the incident stayed valid; some checkouts timed out and had to be retried.

3-D Secure

1. 10:37
   Resolved The queue is clear and delivery is back to normal. If a checkout failed during this window, starting it again issues a fresh code.
2. 10:19
   Monitoring The queue is draining and new codes are going out within seconds. We are holding this open until it is empty.
3. 09:52
   Identified A backlog built up on our outbound mail path after an upstream provider rate-limited us. Messages are queued, not dropped.
4. 09:34
   Investigating Reports of 3-D Secure codes arriving late. Card payments that do not require 3-D Secure are unaffected.

A share of API requests returned 500. The dashboard, card payments and deposits were not affected.

API v1

1. 14:53
   Resolved The rollback is complete and error rates are back to baseline. Failed requests can be retried as-is — no card or deposit was created twice.
2. 14:36
   Identified The cause is a change shipped a few minutes earlier. We are rolling it back.
3. 14:25
   Investigating We are seeing a raised error rate on API requests following a deployment. Investigating now.

Opening a new deposit address took far longer than usual. Deposits already open were unaffected, and no payment was lost.

Crypto deposits

1. 19:30
   Resolved Latency has been stable for fifteen minutes. No deposit was lost and nothing needs to be re-sent.
2. 19:22
   Monitoring Address creation is back to normal timings. We are watching before calling it resolved.
3. 19:08
   Identified One of our payment provider’s upstream nodes is responding slowly. We are routing new requests away from it.
4. 18:56
   Investigating We are seeing elevated latency when reserving a deposit address. Existing deposits and card payments are not affected.