Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.............XXXXXXXXXXX.XXXXXX.............................] duration: 760.188472ms FAIL (2025-08-07 22:48:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 760.212631ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...........................X..................] duration: 1.132746928s FAIL (2025-08-07 22:48:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== g_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214654258 (12h54m14.729756841s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214886232 (12h54m14.729988875s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215184221 (12h54m14.730286807s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215580311 (12h54m14.730682897s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215600089 (12h54m14.730702668s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216063638 (12h54m14.731166221s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216221465 (12h54m14.73132405s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216499611 (12h54m14.73160219s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216784173 (12h54m14.731886753s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216959396 (12h54m14.732061963s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217229150 (12h54m14.732331732s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217364444 (12h54m14.732467028s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217574188 (12h54m14.732676766s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.218700401 (12h54m14.733802981s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.219113148 (12h54m14.734215718s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.307711467 (12h54m14.82281403s) [upload=13273]: Next returned nil segment Aug 7 22:33:40.307743585 (12h54m14.822846164s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:33:40.321114866 (12h54m14.836217441s) [noise=true , address=45.91.116.9:4294 , piece_id=UMCVXJVJ7VWVLOXYVN4KSE5VHM2YXJWOD45364KGN2T62PGBYNOA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.321899105 (12h54m14.837001676s) [noise=true , address=45.91.116.4:4943 , piece_id=4U7YPCLZI7HIZGGBGVWIWJLBRNPGIYOVVUD2UEJJVCMZMC426JIA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.330294085 (12h54m14.845396652s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.331982437 (12h54m14.847085014s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.334537739 (12h54m14.849640313s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.335955579 (12h54m14.851058147s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.340231956 (12h54m14.855334526s) [noise=true , address=77.90.31.11:4559 , piece_id=JSL5C3XIXINYAVAVTQ7J3QD3WWRG6FKP4H4DGJLDOPWCGI7CNAFA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.349513208 (12h54m14.864615777s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.358627538 (12h54m14.873730105s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.436339846 (12h54m14.951442414s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.543248537 (12h54m15.058351106s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.650678621 (12h54m15.165781209s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.786445525 (12h54m15.301548105s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907165087 (12h54m15.422267656s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907202527 (12h54m15.422305104s) [seg_pos={0 0}, upload=13273]: Segment reached optimal threshold of 14 pieces. Aug 7 22:33:40.907372225 (12h54m15.422474801s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QrrGrMHYAjyYiTdTXDUy2uH1j2p1qtUVJaDtais2GowJqBPPu): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.907642442 (12h54m15.422745018s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T): protocol: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection Aug 7 22:33:40.907950578 (12h54m15.423053162s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection Aug 7 22:33:40.908057652 (12h54m15.423160235s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.908080526 (12h54m15.423183103s) [seg_pos={0 0}, upload=13273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:33:40.908212573 (12h54m15.423315155s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:33:40.908218832 (12h54m15.423321415s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:41.213299020 (12h54m15.728401607s) [upload=13273]: Done uploading segments... Aug 7 22:33:41.213326550 (12h54m15.728429128s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:34:48.031458911 (12h55m22.546561477s) [upload=13295]: Starting upload Aug 7 22:34:48.031533867 (12h55m22.546636441s) [upload=13295]: Done starting upload Aug 7 22:34:48.031664339 (12h55m22.546766908s) [upload=13295]: Uploading segments... Aug 7 22:34:48.031895444 (12h55m22.546998028s) [upload=13295]: Got next segment. Inline: false Aug 7 22:34:48.031982233 (12h55m22.547084799s) [upload=13295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:48.031988106 (12h55m22.547090688s) [upload=13295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:34:48.040357077 (12h55m22.555459659s) [upload=13295]: Next segment err: metaclient: Too Many Requests Aug 7 22:34:48.040386485 (12h55m22.555489061s) [upload=13295]: Done uploading segments... Aug 7 22:34:48.040426787 (12h55m22.555529363s) [upload=13295]: Upload finished. err: metaclient: Too Many Requests Duration: 1.132779109s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXX.X............................] duration: 190.596173ms FAIL (2025-08-07 22:48:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g 7 22:20:59.026413148 (12h41m33.541515725s) [upload=13081]: Uploading segments... Aug 7 22:20:59.026487957 (12h41m33.541590544s) [upload=13081]: Got next segment. Inline: true Aug 7 22:20:59.026494660 (12h41m33.54159724s) [upload=13081]: Waiting for error group managing segments... Aug 7 22:20:59.026548283 (12h41m33.541650867s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.026554014 (12h41m33.541656591s) [upload=13081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.026557057 (12h41m33.541659621s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:28.995870666 (12h42m3.510973249s) [upload=13081]: Done uploading segments... Aug 7 22:21:28.995890051 (12h42m3.510992681s) [upload=13081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:29.129769938 (12h42m3.644872515s) [upload=13090]: Starting upload Aug 7 22:21:29.129856429 (12h42m3.644959013s) [upload=13090]: Done starting upload Aug 7 22:21:29.129943698 (12h42m3.645046276s) [upload=13090]: Uploading segments... Aug 7 22:21:29.129985259 (12h42m3.645087845s) [upload=13090]: Got next segment. Inline: true Aug 7 22:21:29.129990476 (12h42m3.645093052s) [upload=13090]: Waiting for error group managing segments... Aug 7 22:21:29.130027566 (12h42m3.645130145s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:29.130031892 (12h42m3.645134469s) [upload=13090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:29.130035507 (12h42m3.645138082s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:59.086555047 (12h42m33.601657631s) [upload=13090]: Done uploading segments... Aug 7 22:21:59.086578788 (12h42m33.601681377s) [upload=13090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:21.787894978 (12h42m56.302997579s) [upload=13102]: Starting upload Aug 7 22:22:21.787975806 (12h42m56.303078386s) [upload=13102]: Done starting upload Aug 7 22:22:21.788049720 (12h42m56.303152297s) [upload=13102]: Uploading segments... Aug 7 22:22:21.788092210 (12h42m56.303194795s) [upload=13102]: Got next segment. Inline: true Aug 7 22:22:21.788097559 (12h42m56.303200126s) [upload=13102]: Waiting for error group managing segments... Aug 7 22:22:21.788141372 (12h42m56.303243949s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:21.788146123 (12h42m56.30324876s) [upload=13102]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:21.788149819 (12h42m56.303252395s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:51.758466888 (12h43m26.273569473s) [upload=13102]: Done uploading segments... Aug 7 22:22:51.758489416 (12h43m26.273591984s) [upload=13102]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:17.561525115 (12h43m52.076627681s) [upload=13117]: Starting upload Aug 7 22:23:17.561618947 (12h43m52.076721524s) [upload=13117]: Done starting upload Aug 7 22:23:17.561741793 (12h43m52.07684436s) [upload=13117]: Uploading segments... Aug 7 22:23:17.561843225 (12h43m52.076945802s) [upload=13117]: Got next segment. Inline: true Aug 7 22:23:17.561848606 (12h43m52.076951175s) [upload=13117]: Waiting for error group managing segments... Aug 7 22:23:17.561909318 (12h43m52.077011885s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:17.561914564 (12h43m52.077017202s) [upload=13117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:17.561918167 (12h43m52.077020738s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:47.532068483 (12h44m22.047171052s) [upload=13117]: Done uploading segments... Aug 7 22:23:47.532095698 (12h44m22.047198268s) [upload=13117]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:55.295508312 (12h44m29.810610893s) [upload=13123]: Starting upload Aug 7 22:23:55.295607041 (12h44m29.81070962s) [upload=13123]: Done starting upload Aug 7 22:23:55.295735905 (12h44m29.810838474s) [upload=13123]: Uploading segments... Aug 7 22:23:55.295799978 (12h44m29.810902554s) [upload=13123]: Got next segment. Inline: true Aug 7 22:23:55.295805637 (12h44m29.810908214s) [upload=13123]: Waiting for error group managing segments... Aug 7 22:23:55.295849955 (12h44m29.810952532s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:55.295854812 (12h44m29.810957387s) [upload=13123]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:55.295858141 (12h44m29.810960717s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:25.257624495 (12h44m59.772727074s) [upload=13123]: Done uploading segments... Aug 7 22:24:25.257647063 (12h44m59.77274964s) [upload=13123]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:51.263752764 (12h45m25.778855407s) [upload=13135]: Starting upload Aug 7 22:24:51.263877660 (12h45m25.778980247s) [upload=13135]: Done starting upload Aug 7 22:24:51.263966776 (12h45m25.779069345s) [upload=13135]: Uploading segments... Aug 7 22:24:51.264018780 (12h45m25.779121365s) [upload=13135]: Got next segment. Inline: true Aug 7 22:24:51.264023895 (12h45m25.779126462s) [upload=13135]: Waiting for error group managing segments... Aug 7 22:24:51.264067265 (12h45m25.779169838s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:51.264097890 (12h45m25.779200467s) [upload=13135]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:51.264101967 (12h45m25.779204534s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:21.220725953 (12h45m55.735828536s) [upload=13135]: Done uploading segments... Aug 7 22:25:21.220748855 (12h45m55.735851436s) [upload=13135]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:25.019445687 (12h45m59.534548303s) [upload=13145]: Starting upload Aug 7 22:25:25.019581109 (12h45m59.534683692s) [upload=13145]: Done starting upload Aug 7 22:25:25.019703288 (12h45m59.534805856s) [upload=13145]: Uploading segments... Aug 7 22:25:25.019823488 (12h45m59.534926072s) [upload=13145]: Got next segment. Inline: true Aug 7 22:25:25.019831359 (12h45m59.534933942s) [upload=13145]: Waiting for error group managing segments... Aug 7 22:25:25.019906993 (12h45m59.535009557s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:25.019929793 (12h45m59.535032371s) [upload=13145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:25.019934887 (12h45m59.535037461s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:54.989474261 (12h46m29.504576827s) [upload=13145]: Done uploading segments... Aug 7 22:25:54.989504165 (12h46m29.50460673s) [upload=13145]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:39.497160053 (12h47m14.012262635s) [upload=13160]: Starting upload Aug 7 22:26:39.497281249 (12h47m14.01238383s) [upload=13160]: Done starting upload Aug 7 22:26:39.497374185 (12h47m14.012476751s) [upload=13160]: Uploading segments... Aug 7 22:26:39.497465038 (12h47m14.012567626s) [upload=13160]: Got next segment. Inline: true Aug 7 22:26:39.497472429 (12h47m14.012575009s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:26:39.497544385 (12h47m14.012646971s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:39.497550182 (12h47m14.012652768s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:39.497554536 (12h47m14.012657133s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.465616489 (12h47m43.980719066s) [upload=13160]: Done uploading segments... Aug 7 22:27:09.465659032 (12h47m43.980761614s) [upload=13160]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:09.584230741 (12h47m44.099333328s) [upload=13164]: Starting upload Aug 7 22:27:09.584314650 (12h47m44.099417234s) [upload=13164]: Done starting upload Aug 7 22:27:09.584433539 (12h47m44.099536113s) [upload=13164]: Uploading segments... Aug 7 22:27:09.584547339 (12h47m44.099649917s) [upload=13164]: Got next segment. Inline: true Aug 7 22:27:09.584576624 (12h47m44.09967921s) [upload=13164]: Waiting for error group managing segments... Aug 7 22:27:09.584764211 (12h47m44.099866798s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:09.584774497 (12h47m44.099877084s) [upload=13164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:09.584779600 (12h47m44.099882182s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.656944239 (12h47m44.172046803s) [upload=13164]: Done uploading segments... Aug 7 22:27:09.656961099 (12h47m44.172063662s) [upload=13164]: Upload finished. err: <nil> Aug 7 22:28:22.451723369 (12h48m56.966825932s) [upload=13182]: Starting upload Aug 7 22:28:22.451781104 (12h48m56.96688368s) [upload=13182]: Done starting upload Aug 7 22:28:22.451842252 (12h48m56.966944827s) [upload=13182]: Uploading segments... Aug 7 22:28:22.451894112 (12h48m56.9669967s) [upload=13182]: Got next segment. Inline: true Aug 7 22:28:22.451901034 (12h48m56.967003616s) [upload=13182]: Waiting for error group managing segments... Aug 7 22:28:22.451944887 (12h48m56.967047449s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:22.451948845 (12h48m56.967051407s) [upload=13182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:22.451951261 (12h48m56.967053837s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:52.423300054 (12h49m26.938402631s) [upload=13182]: Done uploading segments... Aug 7 22:28:52.423324837 (12h49m26.938427414s) [upload=13182]: Upload finished. err: metaclient: context deadline exceeded Duration: 190.6204ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXX...X............................] duration: 537.536585ms FAIL (2025-08-07 22:48:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 537.574388ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXX..X.X..............................] duration: 260.569763ms FAIL (2025-08-07 22:48:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 260.597793ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..............XXXXXXXXXXX.XXXXXX............................] duration: 906.594038ms FAIL (2025-08-07 22:48:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 906.620401ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..XX..................................] duration: 45.985078ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:48:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 46.016343ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................XXXXXXXXX.X...........................] duration: 1.760173178s FAIL (2025-08-07 22:48:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.76020307s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXX.......................] duration: 857.281297ms FAIL (2025-08-07 22:48:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13121]: Uploading piece... Aug 7 22:23:58.634335933 (12h44m33.149438515s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634434236 (12h44m33.149536818s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634966805 (12h44m33.150069389s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634978506 (12h44m33.150081095s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635474598 (12h44m33.150577196s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635575111 (12h44m33.150677682s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635938157 (12h44m33.151040757s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.636367049 (12h44m33.151469627s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.637426749 (12h44m33.152529321s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638866353 (12h44m33.153968932s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638962747 (12h44m33.154065344s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.640916525 (12h44m33.156019112s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642334610 (12h44m33.15743719s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642802034 (12h44m33.157904611s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.644381227 (12h44m33.159483821s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.645186821 (12h44m33.160289426s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.646806610 (12h44m33.161909187s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.722673641 (12h44m33.237776211s) [upload=13121]: Next returned nil segment Aug 7 22:23:58.722691706 (12h44m33.237794289s) [upload=13121]: Waiting for error group managing segments... Aug 7 22:23:58.738782489 (12h44m33.253885056s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.741762463 (12h44m33.25686505s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.744959821 (12h44m33.260062396s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.751545661 (12h44m33.266648241s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.762632925 (12h44m33.277735495s) [noise=true , address=45.91.116.13:4168 , piece_id=5VCOHS7OBO4V2J4FHEQOMMWDK6FCNO7DTOPIURWKUG6XJIZJM2CQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.768246667 (12h44m33.28334923s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770018294 (12h44m33.285120863s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770550717 (12h44m33.285653287s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.772395609 (12h44m33.287498175s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.781308530 (12h44m33.296411093s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.782791410 (12h44m33.297893995s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.875949099 (12h44m33.391051663s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.108952090 (12h44m33.624054668s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158185576 (12h44m33.673288146s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158207517 (12h44m33.673310081s) [seg_pos={0 0}, upload=13121]: Segment reached optimal threshold of 14 pieces. Aug 7 22:23:59.158643218 (12h44m33.6737458s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CiiMtdAoMhPZwSqwraaQcrrx4vZw6L74vadw712VQbRLpNzKN): protocol: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection Aug 7 22:23:59.158730175 (12h44m33.673832737s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection Aug 7 22:23:59.158793697 (12h44m33.673896263s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection Aug 7 22:23:59.158849477 (12h44m33.673952045s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection Aug 7 22:23:59.158861716 (12h44m33.673964283s) [seg_pos={0 0}, upload=13121]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:23:59.158927631 (12h44m33.674030209s) [upload=13121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:23:59.158932133 (12h44m33.6740347s) [upload=13121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:59.182095456 (12h50m33.697198026s) [upload=13121]: Done uploading segments... Aug 7 22:29:59.182119282 (12h50m33.69722185s) [upload=13121]: Upload finished. err: metaclient: context deadline exceeded Duration: 857.312129ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [...............XXXXXXXXXXX.X..X.X...........................] duration: 268.887688ms FAIL (2025-08-07 22:47:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:45.259131358 (12h41m19.774233925s) [upload=13078]: Starting upload Aug 7 22:20:45.259192437 (12h41m19.774295013s) [upload=13078]: Done starting upload Aug 7 22:20:45.259251322 (12h41m19.774353897s) [upload=13078]: Uploading segments... Aug 7 22:20:45.259289829 (12h41m19.774392414s) [upload=13078]: Got next segment. Inline: true Aug 7 22:20:45.259294478 (12h41m19.774397054s) [upload=13078]: Waiting for error group managing segments... Aug 7 22:20:45.259332525 (12h41m19.774435101s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:45.259337069 (12h41m19.774439635s) [upload=13078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:45.259340263 (12h41m19.77444284s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:45.229308083 (12h42m19.744410669s) [upload=13078]: Done uploading segments... Aug 7 22:21:45.229380638 (12h42m19.744483223s) [upload=13078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:59.915154556 (12h42m34.430257139s) [upload=13097]: Starting upload Aug 7 22:21:59.915242531 (12h42m34.430345108s) [upload=13097]: Done starting upload Aug 7 22:21:59.915358515 (12h42m34.430461092s) [upload=13097]: Uploading segments... Aug 7 22:21:59.915398067 (12h42m34.430500653s) [upload=13097]: Got next segment. Inline: true Aug 7 22:21:59.915403151 (12h42m34.430505727s) [upload=13097]: Waiting for error group managing segments... Aug 7 22:21:59.915458296 (12h42m34.430560863s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:59.915463422 (12h42m34.430565998s) [upload=13097]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:59.915466628 (12h42m34.430569269s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.883702618 (12h43m34.398805207s) [upload=13097]: Done uploading segments... Aug 7 22:22:59.883744759 (12h43m34.398847337s) [upload=13097]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:00.217702003 (12h43m34.732804585s) [upload=13110]: Starting upload Aug 7 22:23:00.217805899 (12h43m34.732908483s) [upload=13110]: Done starting upload Aug 7 22:23:00.217896494 (12h43m34.732999072s) [upload=13110]: Uploading segments... Aug 7 22:23:00.217970831 (12h43m34.733073418s) [upload=13110]: Got next segment. Inline: true Aug 7 22:23:00.217979414 (12h43m34.733081991s) [upload=13110]: Waiting for error group managing segments... Aug 7 22:23:00.218043817 (12h43m34.733146394s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:00.218051159 (12h43m34.733153734s) [upload=13110]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:00.218054412 (12h43m34.733156988s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:00.464344185 (12h43m34.979446778s) [upload=13110]: Done uploading segments... Aug 7 22:23:00.464376970 (12h43m34.979479551s) [upload=13110]: Upload finished. err: <nil> Aug 7 22:24:07.143696312 (12h44m41.658798899s) [upload=13125]: Starting upload Aug 7 22:24:07.143811006 (12h44m41.658913589s) [upload=13125]: Done starting upload Aug 7 22:24:07.143914011 (12h44m41.659016577s) [upload=13125]: Uploading segments... Aug 7 22:24:07.143973667 (12h44m41.659076252s) [upload=13125]: Got next segment. Inline: true Aug 7 22:24:07.143979447 (12h44m41.659082034s) [upload=13125]: Waiting for error group managing segments... Aug 7 22:24:07.144045718 (12h44m41.659148297s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:07.144051783 (12h44m41.659154361s) [upload=13125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:07.144057848 (12h44m41.659160429s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.110905816 (12h45m41.626008396s) [upload=13125]: Done uploading segments... Aug 7 22:25:07.110929433 (12h45m41.626032014s) [upload=13125]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:07.223172827 (12h45m41.738275404s) [upload=13138]: Starting upload Aug 7 22:25:07.223254341 (12h45m41.738356918s) [upload=13138]: Done starting upload Aug 7 22:25:07.223368837 (12h45m41.738471414s) [upload=13138]: Uploading segments... Aug 7 22:25:07.223445919 (12h45m41.738548507s) [upload=13138]: Got next segment. Inline: true Aug 7 22:25:07.223452833 (12h45m41.738555412s) [upload=13138]: Waiting for error group managing segments... Aug 7 22:25:07.223511692 (12h45m41.738614278s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:07.223518073 (12h45m41.738620658s) [upload=13138]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:07.223521493 (12h45m41.738624078s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.315623341 (12h45m41.830725924s) [upload=13138]: Done uploading segments... Aug 7 22:25:07.315640785 (12h45m41.830743361s) [upload=13138]: Upload finished. err: <nil> Aug 7 22:26:03.367436251 (12h46m37.882538842s) [upload=13151]: Starting upload Aug 7 22:26:03.367515051 (12h46m37.882617639s) [upload=13151]: Done starting upload Aug 7 22:26:03.367596048 (12h46m37.882698629s) [upload=13151]: Uploading segments... Aug 7 22:26:03.367644909 (12h46m37.8827475s) [upload=13151]: Got next segment. Inline: true Aug 7 22:26:03.367650467 (12h46m37.882753051s) [upload=13151]: Waiting for error group managing segments... Aug 7 22:26:03.367698612 (12h46m37.882801214s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:03.367705011 (12h46m37.882807591s) [upload=13151]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:03.367709618 (12h46m37.882812202s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:03.460311519 (12h46m37.975414085s) [upload=13151]: Done uploading segments... Aug 7 22:26:03.460350352 (12h46m37.975452923s) [upload=13151]: Upload finished. err: <nil> Aug 7 22:26:51.117138988 (12h47m25.632241558s) [upload=13162]: Starting upload Aug 7 22:26:51.117235608 (12h47m25.632338187s) [upload=13162]: Done starting upload Aug 7 22:26:51.117347075 (12h47m25.632449638s) [upload=13162]: Uploading segments... Aug 7 22:26:51.117436995 (12h47m25.632539563s) [upload=13162]: Got next segment. Inline: true Aug 7 22:26:51.117443130 (12h47m25.632545707s) [upload=13162]: Waiting for error group managing segments... Aug 7 22:26:51.117504554 (12h47m25.632607133s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:51.117509417 (12h47m25.632611996s) [upload=13162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:51.117512811 (12h47m25.63261539s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.086979010 (12h48m25.602081586s) [upload=13162]: Done uploading segments... Aug 7 22:27:51.086998672 (12h48m25.602101249s) [upload=13162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:51.193960256 (12h48m25.709062823s) [upload=13178]: Starting upload Aug 7 22:27:51.194024831 (12h48m25.709127407s) [upload=13178]: Done starting upload Aug 7 22:27:51.194084091 (12h48m25.709186657s) [upload=13178]: Uploading segments... Aug 7 22:27:51.194125626 (12h48m25.709228211s) [upload=13178]: Got next segment. Inline: true Aug 7 22:27:51.194132041 (12h48m25.709234607s) [upload=13178]: Waiting for error group managing segments... Aug 7 22:27:51.194172625 (12h48m25.709275192s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:51.194177004 (12h48m25.709279581s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:51.194180164 (12h48m25.709282731s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.254375304 (12h48m25.769477873s) [upload=13178]: Done uploading segments... Aug 7 22:27:51.254404389 (12h48m25.76950696s) [upload=13178]: Upload finished. err: <nil> Aug 7 22:28:44.417417956 (12h49m18.932520542s) [upload=13188]: Starting upload Aug 7 22:28:44.417530389 (12h49m18.932632976s) [upload=13188]: Done starting upload Aug 7 22:28:44.417616643 (12h49m18.93271922s) [upload=13188]: Uploading segments... Aug 7 22:28:44.417757556 (12h49m18.932860142s) [upload=13188]: Got next segment. Inline: true Aug 7 22:28:44.417764563 (12h49m18.932867158s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:28:44.417818209 (12h49m18.932920789s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:44.417825102 (12h49m18.932927678s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:44.417832015 (12h49m18.932934599s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:37.965879533 (12h50m12.480982114s) [upload=13188]: Done uploading segments... Aug 7 22:29:37.965901992 (12h50m12.481004571s) [upload=13188]: Upload finished. err: <nil> Duration: 268.973344ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXX.XX.X.......................] duration: 318.436795ms FAIL (2025-08-07 22:47:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 318.457369ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...............XXXXXXXXXXX.XXXXXX...........................] duration: 867.151061ms FAIL (2025-08-07 22:47:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 867.169279ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXX..X.X.............................] duration: 260.060534ms FAIL (2025-08-07 22:47:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 260.096732ms core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 28.470191ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:47:35 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 28.494022ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXX.X...........................] duration: 253.904279ms FAIL (2025-08-07 22:47:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g 7 22:20:59.026413148 (12h41m33.541515725s) [upload=13081]: Uploading segments... Aug 7 22:20:59.026487957 (12h41m33.541590544s) [upload=13081]: Got next segment. Inline: true Aug 7 22:20:59.026494660 (12h41m33.54159724s) [upload=13081]: Waiting for error group managing segments... Aug 7 22:20:59.026548283 (12h41m33.541650867s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.026554014 (12h41m33.541656591s) [upload=13081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.026557057 (12h41m33.541659621s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:28.995870666 (12h42m3.510973249s) [upload=13081]: Done uploading segments... Aug 7 22:21:28.995890051 (12h42m3.510992681s) [upload=13081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:29.129769938 (12h42m3.644872515s) [upload=13090]: Starting upload Aug 7 22:21:29.129856429 (12h42m3.644959013s) [upload=13090]: Done starting upload Aug 7 22:21:29.129943698 (12h42m3.645046276s) [upload=13090]: Uploading segments... Aug 7 22:21:29.129985259 (12h42m3.645087845s) [upload=13090]: Got next segment. Inline: true Aug 7 22:21:29.129990476 (12h42m3.645093052s) [upload=13090]: Waiting for error group managing segments... Aug 7 22:21:29.130027566 (12h42m3.645130145s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:29.130031892 (12h42m3.645134469s) [upload=13090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:29.130035507 (12h42m3.645138082s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:59.086555047 (12h42m33.601657631s) [upload=13090]: Done uploading segments... Aug 7 22:21:59.086578788 (12h42m33.601681377s) [upload=13090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:21.787894978 (12h42m56.302997579s) [upload=13102]: Starting upload Aug 7 22:22:21.787975806 (12h42m56.303078386s) [upload=13102]: Done starting upload Aug 7 22:22:21.788049720 (12h42m56.303152297s) [upload=13102]: Uploading segments... Aug 7 22:22:21.788092210 (12h42m56.303194795s) [upload=13102]: Got next segment. Inline: true Aug 7 22:22:21.788097559 (12h42m56.303200126s) [upload=13102]: Waiting for error group managing segments... Aug 7 22:22:21.788141372 (12h42m56.303243949s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:21.788146123 (12h42m56.30324876s) [upload=13102]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:21.788149819 (12h42m56.303252395s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:51.758466888 (12h43m26.273569473s) [upload=13102]: Done uploading segments... Aug 7 22:22:51.758489416 (12h43m26.273591984s) [upload=13102]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:17.561525115 (12h43m52.076627681s) [upload=13117]: Starting upload Aug 7 22:23:17.561618947 (12h43m52.076721524s) [upload=13117]: Done starting upload Aug 7 22:23:17.561741793 (12h43m52.07684436s) [upload=13117]: Uploading segments... Aug 7 22:23:17.561843225 (12h43m52.076945802s) [upload=13117]: Got next segment. Inline: true Aug 7 22:23:17.561848606 (12h43m52.076951175s) [upload=13117]: Waiting for error group managing segments... Aug 7 22:23:17.561909318 (12h43m52.077011885s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:17.561914564 (12h43m52.077017202s) [upload=13117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:17.561918167 (12h43m52.077020738s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:47.532068483 (12h44m22.047171052s) [upload=13117]: Done uploading segments... Aug 7 22:23:47.532095698 (12h44m22.047198268s) [upload=13117]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:55.295508312 (12h44m29.810610893s) [upload=13123]: Starting upload Aug 7 22:23:55.295607041 (12h44m29.81070962s) [upload=13123]: Done starting upload Aug 7 22:23:55.295735905 (12h44m29.810838474s) [upload=13123]: Uploading segments... Aug 7 22:23:55.295799978 (12h44m29.810902554s) [upload=13123]: Got next segment. Inline: true Aug 7 22:23:55.295805637 (12h44m29.810908214s) [upload=13123]: Waiting for error group managing segments... Aug 7 22:23:55.295849955 (12h44m29.810952532s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:55.295854812 (12h44m29.810957387s) [upload=13123]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:55.295858141 (12h44m29.810960717s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:25.257624495 (12h44m59.772727074s) [upload=13123]: Done uploading segments... Aug 7 22:24:25.257647063 (12h44m59.77274964s) [upload=13123]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:51.263752764 (12h45m25.778855407s) [upload=13135]: Starting upload Aug 7 22:24:51.263877660 (12h45m25.778980247s) [upload=13135]: Done starting upload Aug 7 22:24:51.263966776 (12h45m25.779069345s) [upload=13135]: Uploading segments... Aug 7 22:24:51.264018780 (12h45m25.779121365s) [upload=13135]: Got next segment. Inline: true Aug 7 22:24:51.264023895 (12h45m25.779126462s) [upload=13135]: Waiting for error group managing segments... Aug 7 22:24:51.264067265 (12h45m25.779169838s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:51.264097890 (12h45m25.779200467s) [upload=13135]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:51.264101967 (12h45m25.779204534s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:21.220725953 (12h45m55.735828536s) [upload=13135]: Done uploading segments... Aug 7 22:25:21.220748855 (12h45m55.735851436s) [upload=13135]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:25.019445687 (12h45m59.534548303s) [upload=13145]: Starting upload Aug 7 22:25:25.019581109 (12h45m59.534683692s) [upload=13145]: Done starting upload Aug 7 22:25:25.019703288 (12h45m59.534805856s) [upload=13145]: Uploading segments... Aug 7 22:25:25.019823488 (12h45m59.534926072s) [upload=13145]: Got next segment. Inline: true Aug 7 22:25:25.019831359 (12h45m59.534933942s) [upload=13145]: Waiting for error group managing segments... Aug 7 22:25:25.019906993 (12h45m59.535009557s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:25.019929793 (12h45m59.535032371s) [upload=13145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:25.019934887 (12h45m59.535037461s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:54.989474261 (12h46m29.504576827s) [upload=13145]: Done uploading segments... Aug 7 22:25:54.989504165 (12h46m29.50460673s) [upload=13145]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:39.497160053 (12h47m14.012262635s) [upload=13160]: Starting upload Aug 7 22:26:39.497281249 (12h47m14.01238383s) [upload=13160]: Done starting upload Aug 7 22:26:39.497374185 (12h47m14.012476751s) [upload=13160]: Uploading segments... Aug 7 22:26:39.497465038 (12h47m14.012567626s) [upload=13160]: Got next segment. Inline: true Aug 7 22:26:39.497472429 (12h47m14.012575009s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:26:39.497544385 (12h47m14.012646971s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:39.497550182 (12h47m14.012652768s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:39.497554536 (12h47m14.012657133s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.465616489 (12h47m43.980719066s) [upload=13160]: Done uploading segments... Aug 7 22:27:09.465659032 (12h47m43.980761614s) [upload=13160]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:09.584230741 (12h47m44.099333328s) [upload=13164]: Starting upload Aug 7 22:27:09.584314650 (12h47m44.099417234s) [upload=13164]: Done starting upload Aug 7 22:27:09.584433539 (12h47m44.099536113s) [upload=13164]: Uploading segments... Aug 7 22:27:09.584547339 (12h47m44.099649917s) [upload=13164]: Got next segment. Inline: true Aug 7 22:27:09.584576624 (12h47m44.09967921s) [upload=13164]: Waiting for error group managing segments... Aug 7 22:27:09.584764211 (12h47m44.099866798s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:09.584774497 (12h47m44.099877084s) [upload=13164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:09.584779600 (12h47m44.099882182s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.656944239 (12h47m44.172046803s) [upload=13164]: Done uploading segments... Aug 7 22:27:09.656961099 (12h47m44.172063662s) [upload=13164]: Upload finished. err: <nil> Aug 7 22:28:22.451723369 (12h48m56.966825932s) [upload=13182]: Starting upload Aug 7 22:28:22.451781104 (12h48m56.96688368s) [upload=13182]: Done starting upload Aug 7 22:28:22.451842252 (12h48m56.966944827s) [upload=13182]: Uploading segments... Aug 7 22:28:22.451894112 (12h48m56.9669967s) [upload=13182]: Got next segment. Inline: true Aug 7 22:28:22.451901034 (12h48m56.967003616s) [upload=13182]: Waiting for error group managing segments... Aug 7 22:28:22.451944887 (12h48m56.967047449s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:22.451948845 (12h48m56.967051407s) [upload=13182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:22.451951261 (12h48m56.967053837s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:52.423300054 (12h49m26.938402631s) [upload=13182]: Done uploading segments... Aug 7 22:28:52.423324837 (12h49m26.938427414s) [upload=13182]: Upload finished. err: metaclient: context deadline exceeded Duration: 253.997541ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..........................XXXXXXXXXXX......................] duration: 1.382665414s FAIL (2025-08-07 22:47:30 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13121]: Uploading piece... Aug 7 22:23:58.634335933 (12h44m33.149438515s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634434236 (12h44m33.149536818s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634966805 (12h44m33.150069389s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634978506 (12h44m33.150081095s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635474598 (12h44m33.150577196s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635575111 (12h44m33.150677682s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635938157 (12h44m33.151040757s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.636367049 (12h44m33.151469627s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.637426749 (12h44m33.152529321s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638866353 (12h44m33.153968932s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638962747 (12h44m33.154065344s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.640916525 (12h44m33.156019112s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642334610 (12h44m33.15743719s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642802034 (12h44m33.157904611s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.644381227 (12h44m33.159483821s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.645186821 (12h44m33.160289426s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.646806610 (12h44m33.161909187s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.722673641 (12h44m33.237776211s) [upload=13121]: Next returned nil segment Aug 7 22:23:58.722691706 (12h44m33.237794289s) [upload=13121]: Waiting for error group managing segments... Aug 7 22:23:58.738782489 (12h44m33.253885056s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.741762463 (12h44m33.25686505s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.744959821 (12h44m33.260062396s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.751545661 (12h44m33.266648241s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.762632925 (12h44m33.277735495s) [noise=true , address=45.91.116.13:4168 , piece_id=5VCOHS7OBO4V2J4FHEQOMMWDK6FCNO7DTOPIURWKUG6XJIZJM2CQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.768246667 (12h44m33.28334923s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770018294 (12h44m33.285120863s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770550717 (12h44m33.285653287s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.772395609 (12h44m33.287498175s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.781308530 (12h44m33.296411093s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.782791410 (12h44m33.297893995s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.875949099 (12h44m33.391051663s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.108952090 (12h44m33.624054668s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158185576 (12h44m33.673288146s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158207517 (12h44m33.673310081s) [seg_pos={0 0}, upload=13121]: Segment reached optimal threshold of 14 pieces. Aug 7 22:23:59.158643218 (12h44m33.6737458s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CiiMtdAoMhPZwSqwraaQcrrx4vZw6L74vadw712VQbRLpNzKN): protocol: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection Aug 7 22:23:59.158730175 (12h44m33.673832737s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection Aug 7 22:23:59.158793697 (12h44m33.673896263s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection Aug 7 22:23:59.158849477 (12h44m33.673952045s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection Aug 7 22:23:59.158861716 (12h44m33.673964283s) [seg_pos={0 0}, upload=13121]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:23:59.158927631 (12h44m33.674030209s) [upload=13121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:23:59.158932133 (12h44m33.6740347s) [upload=13121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:59.182095456 (12h50m33.697198026s) [upload=13121]: Done uploading segments... Aug 7 22:29:59.182119282 (12h50m33.69722185s) [upload=13121]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.382692512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXX..X.X............................] duration: 241.045278ms FAIL (2025-08-07 22:47:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 241.071144ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX.........................X.XX...........................] duration: 884.567369ms FAIL (2025-08-07 22:47:25 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=12957]: Uploading segments... Aug 7 22:12:11.442470431 (12h32m45.957573001s) [upload=12957]: Got next segment. Inline: false Aug 7 22:12:11.442536399 (12h32m45.957638969s) [upload=12957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.442542497 (12h32m45.957645072s) [upload=12957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:41.409489142 (12h33m15.924591718s) [upload=12957]: Next segment err: context deadline exceeded Aug 7 22:12:41.409585580 (12h33m15.924688156s) [upload=12957]: Done uploading segments... Aug 7 22:12:41.409635672 (12h33m15.92473824s) [upload=12957]: Upload finished. err: context deadline exceeded Aug 7 22:12:41.517765572 (12h33m16.032868143s) [upload=12968]: Starting upload Aug 7 22:12:41.517838005 (12h33m16.032940574s) [upload=12968]: Done starting upload Aug 7 22:12:41.517963005 (12h33m16.033065573s) [upload=12968]: Uploading segments... Aug 7 22:12:41.518335606 (12h33m16.033438184s) [upload=12968]: Got next segment. Inline: false Aug 7 22:12:41.518426941 (12h33m16.033529524s) [upload=12968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:41.518434836 (12h33m16.03353742s) [upload=12968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:48.560277241 (12h33m23.075379809s) [seg_pos={0 0}, upload=12968]: Begin upload segment... Aug 7 22:12:48.560321663 (12h33m23.075424233s) [seg_pos={0 0}, upload=12968]: Done begin upload segment. Aug 7 22:12:48.560346254 (12h33m23.075448833s) [noise=true , address=77.90.31.21:4878 , piece_id=AYEE7NAJIYWU72ZKPBR44ZRU7T7HUB4FPAHH6V3KYKBSULV3AJMQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.560385235 (12h33m23.075487817s) [noise=true , address=104.36.82.4:4802 , piece_id=WUKRWI3KJKXS2KBP5WRMPI2UGYT7RKQ6KBBOAPPW6Z5ISN72VIWQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.560895927 (12h33m23.075998514s) [noise=true , address=45.91.116.5:4297 , piece_id=5TO5AKA4MGTPNPH6VWPA4GIDECUKKPFWSR5VZ3263TYANZ4BZGPA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.561270415 (12h33m23.076372997s) [noise=true , address=45.91.116.9:4262 , piece_id=UGR7YOFEX6MXTFNZPJ7EKO5DH3V7R47FOKI5ZNAHB4ZYH7J7UHKA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.562228406 (12h33m23.077330973s) [noise=true , address=45.91.116.11:4117 , piece_id=PQYC55OVQBT5EFFWHBMXXDFBGCDNBZFAYJR42IK5JEDDAXXWDJYA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.563480004 (12h33m23.078582611s) [noise=true , address=45.91.116.14:4190 , piece_id=3U7NYGD2JPOBZAHNJ2GUNUA3GE6JYRLJBMYQTEREUVWBR6QWAM6A, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.563854062 (12h33m23.078956648s) [noise=true , address=45.91.116.11:4118 , piece_id=A5SGTLAHM6E3N7AHB7U3KGZFYHDKO4JBE2IHNH76YKJUR54TGMZA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.564192678 (12h33m23.079295261s) [noise=true , address=104.36.82.12:4151 , piece_id=BV2GWAIQGUQ3BKVWHDBQ2RK772VQTLGYJ4WZU5VPALAGO7VOAZMA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.564542455 (12h33m23.079645037s) [noise=true , address=77.90.31.17:4771 , piece_id=OZMRVKETNTDWOLXUFP3IJLGQENMWSWB3MOTJKXXRUEDQHOFV6LQA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565038234 (12h33m23.080140824s) [noise=true , address=77.90.31.16:4733 , piece_id=LB4CQZC5X7QUEYL5OAQ4A4MA6OFO6LPC7RKCLN5AT6V6UGFHAKEQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565530690 (12h33m23.08063327s) [noise=true , address=77.90.31.10:4520 , piece_id=BKGMZCYXKDUXT42DSCXXLBKMYASXQWCG43HN4NJT76A5EESUNI3Q, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565683172 (12h33m23.080785758s) [noise=true , address=45.91.116.6:4975 , piece_id=KALV4QK4ML56PQUNJFDQXHTXZ5VOJYWZMXQNGHA6U5FLBXEW46TA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566145847 (12h33m23.081248429s) [noise=true , address=77.90.31.11:4553 , piece_id=TY7MWYZNDLY7CMDYALZFEYN2HE64TUEYW522DJKBFL463MMQIPOA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566452821 (12h33m23.0815554s) [noise=true , address=77.90.31.18:4805 , piece_id=NP4IZUJHXPID26UBGUMTHYRM2TEDF2BHPAE5Z4B34OVHJ4RRX5LA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566765755 (12h33m23.081868318s) [noise=true , address=77.90.31.21:4877 , piece_id=53XKORZS4TOBNGN2VM7DF55MS4NCZH7NX7F4ZZHDMJCKNT3YPVZQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.567074238 (12h33m23.082176801s) [noise=true , address=104.36.82.6:4876 , piece_id=MFX6SCYKRA4PAZ4VJ3JMKK6THX5MOCE4SBENR6NXDLOWA7C2GGJQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.674430176 (12h33m23.189532754s) [upload=12968]: Next returned nil segment Aug 7 22:12:48.674443729 (12h33m23.189546305s) [upload=12968]: Waiting for error group managing segments... Aug 7 22:12:48.677024873 (12h33m23.192127438s) [noise=true , address=77.90.31.11:4553 , piece_id=TY7MWYZNDLY7CMDYALZFEYN2HE64TUEYW522DJKBFL463MMQIPOA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.677316512 (12h33m23.192419097s) [noise=true , address=45.91.116.5:4297 , piece_id=5TO5AKA4MGTPNPH6VWPA4GIDECUKKPFWSR5VZ3263TYANZ4BZGPA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.677774329 (12h33m23.192876899s) [noise=true , address=45.91.116.9:4262 , piece_id=UGR7YOFEX6MXTFNZPJ7EKO5DH3V7R47FOKI5ZNAHB4ZYH7J7UHKA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678157641 (12h33m23.193260219s) [noise=true , address=45.91.116.11:4118 , piece_id=A5SGTLAHM6E3N7AHB7U3KGZFYHDKO4JBE2IHNH76YKJUR54TGMZA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678319308 (12h33m23.193421874s) [noise=true , address=45.91.116.11:4117 , piece_id=PQYC55OVQBT5EFFWHBMXXDFBGCDNBZFAYJR42IK5JEDDAXXWDJYA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678828950 (12h33m23.193931524s) [noise=true , address=45.91.116.6:4975 , piece_id=KALV4QK4ML56PQUNJFDQXHTXZ5VOJYWZMXQNGHA6U5FLBXEW46TA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678921127 (12h33m23.194023707s) [noise=true , address=45.91.116.14:4190 , piece_id=3U7NYGD2JPOBZAHNJ2GUNUA3GE6JYRLJBMYQTEREUVWBR6QWAM6A, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.679383015 (12h33m23.194485578s) [noise=true , address=77.90.31.21:4877 , piece_id=53XKORZS4TOBNGN2VM7DF55MS4NCZH7NX7F4ZZHDMJCKNT3YPVZQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.679808178 (12h33m23.194910761s) [noise=true , address=77.90.31.18:4805 , piece_id=NP4IZUJHXPID26UBGUMTHYRM2TEDF2BHPAE5Z4B34OVHJ4RRX5LA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.680957829 (12h33m23.196060398s) [noise=true , address=77.90.31.21:4878 , piece_id=AYEE7NAJIYWU72ZKPBR44ZRU7T7HUB4FPAHH6V3KYKBSULV3AJMQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.681390098 (12h33m23.196492675s) [noise=true , address=104.36.82.4:4802 , piece_id=WUKRWI3KJKXS2KBP5WRMPI2UGYT7RKQ6KBBOAPPW6Z5ISN72VIWQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.681868182 (12h33m23.19697076s) [noise=true , address=77.90.31.16:4733 , piece_id=LB4CQZC5X7QUEYL5OAQ4A4MA6OFO6LPC7RKCLN5AT6V6UGFHAKEQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.682434566 (12h33m23.197537204s) [noise=true , address=77.90.31.17:4771 , piece_id=OZMRVKETNTDWOLXUFP3IJLGQENMWSWB3MOTJKXXRUEDQHOFV6LQA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.682445484 (12h33m23.197548047s) [seg_pos={0 0}, upload=12968]: Segment reached optimal threshold of 13 pieces. Aug 7 22:12:48.682659606 (12h33m23.197762177s) [noise=true , address=77.90.31.10:4520 , piece_id=BKGMZCYXKDUXT42DSCXXLBKMYASXQWCG43HN4NJT76A5EESUNI3Q, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:48.682743670 (12h33m23.197846239s) [noise=true , address=104.36.82.12:4151 , piece_id=BV2GWAIQGUQ3BKVWHDBQ2RK772VQTLGYJ4WZU5VPALAGO7VOAZMA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection; write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection Aug 7 22:12:48.682881386 (12h33m23.197983962s) [noise=true , address=104.36.82.6:4876 , piece_id=MFX6SCYKRA4PAZ4VJ3JMKK6THX5MOCE4SBENR6NXDLOWA7C2GGJQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: read tcp 159.223.143.39:35642->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 22:12:48.682925198 (12h33m23.198027778s) [seg_pos={0 0}, upload=12968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:12:48.683071518 (12h33m23.198174098s) [upload=12968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:48.683078012 (12h33m23.19818059s) [upload=12968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:26.914480867 (12h50m1.429583435s) [upload=12968]: Done uploading segments... Aug 7 22:29:26.914503234 (12h50m1.42960581s) [upload=12968]: Upload finished. err: metaclient: context deadline exceeded Duration: 884.598341ms core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [......................X..XX.................................] duration: 42.778251ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:47:22 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 42.802362ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [................XXXXXXXXXXX.X..X.X..........................] duration: 210.257075ms FAIL (2025-08-07 22:47:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:45.259131358 (12h41m19.774233925s) [upload=13078]: Starting upload Aug 7 22:20:45.259192437 (12h41m19.774295013s) [upload=13078]: Done starting upload Aug 7 22:20:45.259251322 (12h41m19.774353897s) [upload=13078]: Uploading segments... Aug 7 22:20:45.259289829 (12h41m19.774392414s) [upload=13078]: Got next segment. Inline: true Aug 7 22:20:45.259294478 (12h41m19.774397054s) [upload=13078]: Waiting for error group managing segments... Aug 7 22:20:45.259332525 (12h41m19.774435101s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:45.259337069 (12h41m19.774439635s) [upload=13078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:45.259340263 (12h41m19.77444284s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:45.229308083 (12h42m19.744410669s) [upload=13078]: Done uploading segments... Aug 7 22:21:45.229380638 (12h42m19.744483223s) [upload=13078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:59.915154556 (12h42m34.430257139s) [upload=13097]: Starting upload Aug 7 22:21:59.915242531 (12h42m34.430345108s) [upload=13097]: Done starting upload Aug 7 22:21:59.915358515 (12h42m34.430461092s) [upload=13097]: Uploading segments... Aug 7 22:21:59.915398067 (12h42m34.430500653s) [upload=13097]: Got next segment. Inline: true Aug 7 22:21:59.915403151 (12h42m34.430505727s) [upload=13097]: Waiting for error group managing segments... Aug 7 22:21:59.915458296 (12h42m34.430560863s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:59.915463422 (12h42m34.430565998s) [upload=13097]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:59.915466628 (12h42m34.430569269s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.883702618 (12h43m34.398805207s) [upload=13097]: Done uploading segments... Aug 7 22:22:59.883744759 (12h43m34.398847337s) [upload=13097]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:00.217702003 (12h43m34.732804585s) [upload=13110]: Starting upload Aug 7 22:23:00.217805899 (12h43m34.732908483s) [upload=13110]: Done starting upload Aug 7 22:23:00.217896494 (12h43m34.732999072s) [upload=13110]: Uploading segments... Aug 7 22:23:00.217970831 (12h43m34.733073418s) [upload=13110]: Got next segment. Inline: true Aug 7 22:23:00.217979414 (12h43m34.733081991s) [upload=13110]: Waiting for error group managing segments... Aug 7 22:23:00.218043817 (12h43m34.733146394s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:00.218051159 (12h43m34.733153734s) [upload=13110]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:00.218054412 (12h43m34.733156988s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:00.464344185 (12h43m34.979446778s) [upload=13110]: Done uploading segments... Aug 7 22:23:00.464376970 (12h43m34.979479551s) [upload=13110]: Upload finished. err: <nil> Aug 7 22:24:07.143696312 (12h44m41.658798899s) [upload=13125]: Starting upload Aug 7 22:24:07.143811006 (12h44m41.658913589s) [upload=13125]: Done starting upload Aug 7 22:24:07.143914011 (12h44m41.659016577s) [upload=13125]: Uploading segments... Aug 7 22:24:07.143973667 (12h44m41.659076252s) [upload=13125]: Got next segment. Inline: true Aug 7 22:24:07.143979447 (12h44m41.659082034s) [upload=13125]: Waiting for error group managing segments... Aug 7 22:24:07.144045718 (12h44m41.659148297s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:07.144051783 (12h44m41.659154361s) [upload=13125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:07.144057848 (12h44m41.659160429s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.110905816 (12h45m41.626008396s) [upload=13125]: Done uploading segments... Aug 7 22:25:07.110929433 (12h45m41.626032014s) [upload=13125]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:07.223172827 (12h45m41.738275404s) [upload=13138]: Starting upload Aug 7 22:25:07.223254341 (12h45m41.738356918s) [upload=13138]: Done starting upload Aug 7 22:25:07.223368837 (12h45m41.738471414s) [upload=13138]: Uploading segments... Aug 7 22:25:07.223445919 (12h45m41.738548507s) [upload=13138]: Got next segment. Inline: true Aug 7 22:25:07.223452833 (12h45m41.738555412s) [upload=13138]: Waiting for error group managing segments... Aug 7 22:25:07.223511692 (12h45m41.738614278s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:07.223518073 (12h45m41.738620658s) [upload=13138]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:07.223521493 (12h45m41.738624078s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.315623341 (12h45m41.830725924s) [upload=13138]: Done uploading segments... Aug 7 22:25:07.315640785 (12h45m41.830743361s) [upload=13138]: Upload finished. err: <nil> Aug 7 22:26:03.367436251 (12h46m37.882538842s) [upload=13151]: Starting upload Aug 7 22:26:03.367515051 (12h46m37.882617639s) [upload=13151]: Done starting upload Aug 7 22:26:03.367596048 (12h46m37.882698629s) [upload=13151]: Uploading segments... Aug 7 22:26:03.367644909 (12h46m37.8827475s) [upload=13151]: Got next segment. Inline: true Aug 7 22:26:03.367650467 (12h46m37.882753051s) [upload=13151]: Waiting for error group managing segments... Aug 7 22:26:03.367698612 (12h46m37.882801214s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:03.367705011 (12h46m37.882807591s) [upload=13151]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:03.367709618 (12h46m37.882812202s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:03.460311519 (12h46m37.975414085s) [upload=13151]: Done uploading segments... Aug 7 22:26:03.460350352 (12h46m37.975452923s) [upload=13151]: Upload finished. err: <nil> Aug 7 22:26:51.117138988 (12h47m25.632241558s) [upload=13162]: Starting upload Aug 7 22:26:51.117235608 (12h47m25.632338187s) [upload=13162]: Done starting upload Aug 7 22:26:51.117347075 (12h47m25.632449638s) [upload=13162]: Uploading segments... Aug 7 22:26:51.117436995 (12h47m25.632539563s) [upload=13162]: Got next segment. Inline: true Aug 7 22:26:51.117443130 (12h47m25.632545707s) [upload=13162]: Waiting for error group managing segments... Aug 7 22:26:51.117504554 (12h47m25.632607133s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:51.117509417 (12h47m25.632611996s) [upload=13162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:51.117512811 (12h47m25.63261539s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.086979010 (12h48m25.602081586s) [upload=13162]: Done uploading segments... Aug 7 22:27:51.086998672 (12h48m25.602101249s) [upload=13162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:51.193960256 (12h48m25.709062823s) [upload=13178]: Starting upload Aug 7 22:27:51.194024831 (12h48m25.709127407s) [upload=13178]: Done starting upload Aug 7 22:27:51.194084091 (12h48m25.709186657s) [upload=13178]: Uploading segments... Aug 7 22:27:51.194125626 (12h48m25.709228211s) [upload=13178]: Got next segment. Inline: true Aug 7 22:27:51.194132041 (12h48m25.709234607s) [upload=13178]: Waiting for error group managing segments... Aug 7 22:27:51.194172625 (12h48m25.709275192s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:51.194177004 (12h48m25.709279581s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:51.194180164 (12h48m25.709282731s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.254375304 (12h48m25.769477873s) [upload=13178]: Done uploading segments... Aug 7 22:27:51.254404389 (12h48m25.76950696s) [upload=13178]: Upload finished. err: <nil> Aug 7 22:28:44.417417956 (12h49m18.932520542s) [upload=13188]: Starting upload Aug 7 22:28:44.417530389 (12h49m18.932632976s) [upload=13188]: Done starting upload Aug 7 22:28:44.417616643 (12h49m18.93271922s) [upload=13188]: Uploading segments... Aug 7 22:28:44.417757556 (12h49m18.932860142s) [upload=13188]: Got next segment. Inline: true Aug 7 22:28:44.417764563 (12h49m18.932867158s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:28:44.417818209 (12h49m18.932920789s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:44.417825102 (12h49m18.932927678s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:44.417832015 (12h49m18.932934599s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:37.965879533 (12h50m12.480982114s) [upload=13188]: Done uploading segments... Aug 7 22:29:37.965901992 (12h50m12.481004571s) [upload=13188]: Upload finished. err: <nil> Duration: 210.286022ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...........................X.................] duration: 798.68124ms FAIL (2025-08-07 22:47:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== g_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214654258 (12h54m14.729756841s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214886232 (12h54m14.729988875s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215184221 (12h54m14.730286807s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215580311 (12h54m14.730682897s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215600089 (12h54m14.730702668s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216063638 (12h54m14.731166221s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216221465 (12h54m14.73132405s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216499611 (12h54m14.73160219s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216784173 (12h54m14.731886753s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216959396 (12h54m14.732061963s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217229150 (12h54m14.732331732s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217364444 (12h54m14.732467028s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217574188 (12h54m14.732676766s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.218700401 (12h54m14.733802981s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.219113148 (12h54m14.734215718s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.307711467 (12h54m14.82281403s) [upload=13273]: Next returned nil segment Aug 7 22:33:40.307743585 (12h54m14.822846164s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:33:40.321114866 (12h54m14.836217441s) [noise=true , address=45.91.116.9:4294 , piece_id=UMCVXJVJ7VWVLOXYVN4KSE5VHM2YXJWOD45364KGN2T62PGBYNOA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.321899105 (12h54m14.837001676s) [noise=true , address=45.91.116.4:4943 , piece_id=4U7YPCLZI7HIZGGBGVWIWJLBRNPGIYOVVUD2UEJJVCMZMC426JIA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.330294085 (12h54m14.845396652s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.331982437 (12h54m14.847085014s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.334537739 (12h54m14.849640313s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.335955579 (12h54m14.851058147s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.340231956 (12h54m14.855334526s) [noise=true , address=77.90.31.11:4559 , piece_id=JSL5C3XIXINYAVAVTQ7J3QD3WWRG6FKP4H4DGJLDOPWCGI7CNAFA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.349513208 (12h54m14.864615777s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.358627538 (12h54m14.873730105s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.436339846 (12h54m14.951442414s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.543248537 (12h54m15.058351106s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.650678621 (12h54m15.165781209s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.786445525 (12h54m15.301548105s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907165087 (12h54m15.422267656s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907202527 (12h54m15.422305104s) [seg_pos={0 0}, upload=13273]: Segment reached optimal threshold of 14 pieces. Aug 7 22:33:40.907372225 (12h54m15.422474801s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QrrGrMHYAjyYiTdTXDUy2uH1j2p1qtUVJaDtais2GowJqBPPu): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.907642442 (12h54m15.422745018s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T): protocol: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection Aug 7 22:33:40.907950578 (12h54m15.423053162s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection Aug 7 22:33:40.908057652 (12h54m15.423160235s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.908080526 (12h54m15.423183103s) [seg_pos={0 0}, upload=13273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:33:40.908212573 (12h54m15.423315155s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:33:40.908218832 (12h54m15.423321415s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:41.213299020 (12h54m15.728401607s) [upload=13273]: Done uploading segments... Aug 7 22:33:41.213326550 (12h54m15.728429128s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:34:48.031458911 (12h55m22.546561477s) [upload=13295]: Starting upload Aug 7 22:34:48.031533867 (12h55m22.546636441s) [upload=13295]: Done starting upload Aug 7 22:34:48.031664339 (12h55m22.546766908s) [upload=13295]: Uploading segments... Aug 7 22:34:48.031895444 (12h55m22.546998028s) [upload=13295]: Got next segment. Inline: false Aug 7 22:34:48.031982233 (12h55m22.547084799s) [upload=13295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:48.031988106 (12h55m22.547090688s) [upload=13295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:34:48.040357077 (12h55m22.555459659s) [upload=13295]: Next segment err: metaclient: Too Many Requests Aug 7 22:34:48.040386485 (12h55m22.555489061s) [upload=13295]: Done uploading segments... Aug 7 22:34:48.040426787 (12h55m22.555529363s) [upload=13295]: Upload finished. err: metaclient: Too Many Requests Duration: 798.716049ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 26.017588ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:47:14 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 26.044879ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXX...X...........................] duration: 547.07587ms FAIL (2025-08-07 22:47:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 547.097599ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................XXXXXXXXX.X..........................] duration: 1.67870106s FAIL (2025-08-07 22:47:10 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.678725551s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [.................XXXXXXXXXXX.X..X.X.........................] duration: 229.224197ms FAIL (2025-08-07 22:47:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:45.259131358 (12h41m19.774233925s) [upload=13078]: Starting upload Aug 7 22:20:45.259192437 (12h41m19.774295013s) [upload=13078]: Done starting upload Aug 7 22:20:45.259251322 (12h41m19.774353897s) [upload=13078]: Uploading segments... Aug 7 22:20:45.259289829 (12h41m19.774392414s) [upload=13078]: Got next segment. Inline: true Aug 7 22:20:45.259294478 (12h41m19.774397054s) [upload=13078]: Waiting for error group managing segments... Aug 7 22:20:45.259332525 (12h41m19.774435101s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:45.259337069 (12h41m19.774439635s) [upload=13078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:45.259340263 (12h41m19.77444284s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:45.229308083 (12h42m19.744410669s) [upload=13078]: Done uploading segments... Aug 7 22:21:45.229380638 (12h42m19.744483223s) [upload=13078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:59.915154556 (12h42m34.430257139s) [upload=13097]: Starting upload Aug 7 22:21:59.915242531 (12h42m34.430345108s) [upload=13097]: Done starting upload Aug 7 22:21:59.915358515 (12h42m34.430461092s) [upload=13097]: Uploading segments... Aug 7 22:21:59.915398067 (12h42m34.430500653s) [upload=13097]: Got next segment. Inline: true Aug 7 22:21:59.915403151 (12h42m34.430505727s) [upload=13097]: Waiting for error group managing segments... Aug 7 22:21:59.915458296 (12h42m34.430560863s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:59.915463422 (12h42m34.430565998s) [upload=13097]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:59.915466628 (12h42m34.430569269s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.883702618 (12h43m34.398805207s) [upload=13097]: Done uploading segments... Aug 7 22:22:59.883744759 (12h43m34.398847337s) [upload=13097]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:00.217702003 (12h43m34.732804585s) [upload=13110]: Starting upload Aug 7 22:23:00.217805899 (12h43m34.732908483s) [upload=13110]: Done starting upload Aug 7 22:23:00.217896494 (12h43m34.732999072s) [upload=13110]: Uploading segments... Aug 7 22:23:00.217970831 (12h43m34.733073418s) [upload=13110]: Got next segment. Inline: true Aug 7 22:23:00.217979414 (12h43m34.733081991s) [upload=13110]: Waiting for error group managing segments... Aug 7 22:23:00.218043817 (12h43m34.733146394s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:00.218051159 (12h43m34.733153734s) [upload=13110]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:00.218054412 (12h43m34.733156988s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:00.464344185 (12h43m34.979446778s) [upload=13110]: Done uploading segments... Aug 7 22:23:00.464376970 (12h43m34.979479551s) [upload=13110]: Upload finished. err: <nil> Aug 7 22:24:07.143696312 (12h44m41.658798899s) [upload=13125]: Starting upload Aug 7 22:24:07.143811006 (12h44m41.658913589s) [upload=13125]: Done starting upload Aug 7 22:24:07.143914011 (12h44m41.659016577s) [upload=13125]: Uploading segments... Aug 7 22:24:07.143973667 (12h44m41.659076252s) [upload=13125]: Got next segment. Inline: true Aug 7 22:24:07.143979447 (12h44m41.659082034s) [upload=13125]: Waiting for error group managing segments... Aug 7 22:24:07.144045718 (12h44m41.659148297s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:07.144051783 (12h44m41.659154361s) [upload=13125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:07.144057848 (12h44m41.659160429s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.110905816 (12h45m41.626008396s) [upload=13125]: Done uploading segments... Aug 7 22:25:07.110929433 (12h45m41.626032014s) [upload=13125]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:07.223172827 (12h45m41.738275404s) [upload=13138]: Starting upload Aug 7 22:25:07.223254341 (12h45m41.738356918s) [upload=13138]: Done starting upload Aug 7 22:25:07.223368837 (12h45m41.738471414s) [upload=13138]: Uploading segments... Aug 7 22:25:07.223445919 (12h45m41.738548507s) [upload=13138]: Got next segment. Inline: true Aug 7 22:25:07.223452833 (12h45m41.738555412s) [upload=13138]: Waiting for error group managing segments... Aug 7 22:25:07.223511692 (12h45m41.738614278s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:07.223518073 (12h45m41.738620658s) [upload=13138]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:07.223521493 (12h45m41.738624078s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.315623341 (12h45m41.830725924s) [upload=13138]: Done uploading segments... Aug 7 22:25:07.315640785 (12h45m41.830743361s) [upload=13138]: Upload finished. err: <nil> Aug 7 22:26:03.367436251 (12h46m37.882538842s) [upload=13151]: Starting upload Aug 7 22:26:03.367515051 (12h46m37.882617639s) [upload=13151]: Done starting upload Aug 7 22:26:03.367596048 (12h46m37.882698629s) [upload=13151]: Uploading segments... Aug 7 22:26:03.367644909 (12h46m37.8827475s) [upload=13151]: Got next segment. Inline: true Aug 7 22:26:03.367650467 (12h46m37.882753051s) [upload=13151]: Waiting for error group managing segments... Aug 7 22:26:03.367698612 (12h46m37.882801214s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:03.367705011 (12h46m37.882807591s) [upload=13151]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:03.367709618 (12h46m37.882812202s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:03.460311519 (12h46m37.975414085s) [upload=13151]: Done uploading segments... Aug 7 22:26:03.460350352 (12h46m37.975452923s) [upload=13151]: Upload finished. err: <nil> Aug 7 22:26:51.117138988 (12h47m25.632241558s) [upload=13162]: Starting upload Aug 7 22:26:51.117235608 (12h47m25.632338187s) [upload=13162]: Done starting upload Aug 7 22:26:51.117347075 (12h47m25.632449638s) [upload=13162]: Uploading segments... Aug 7 22:26:51.117436995 (12h47m25.632539563s) [upload=13162]: Got next segment. Inline: true Aug 7 22:26:51.117443130 (12h47m25.632545707s) [upload=13162]: Waiting for error group managing segments... Aug 7 22:26:51.117504554 (12h47m25.632607133s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:51.117509417 (12h47m25.632611996s) [upload=13162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:51.117512811 (12h47m25.63261539s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.086979010 (12h48m25.602081586s) [upload=13162]: Done uploading segments... Aug 7 22:27:51.086998672 (12h48m25.602101249s) [upload=13162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:51.193960256 (12h48m25.709062823s) [upload=13178]: Starting upload Aug 7 22:27:51.194024831 (12h48m25.709127407s) [upload=13178]: Done starting upload Aug 7 22:27:51.194084091 (12h48m25.709186657s) [upload=13178]: Uploading segments... Aug 7 22:27:51.194125626 (12h48m25.709228211s) [upload=13178]: Got next segment. Inline: true Aug 7 22:27:51.194132041 (12h48m25.709234607s) [upload=13178]: Waiting for error group managing segments... Aug 7 22:27:51.194172625 (12h48m25.709275192s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:51.194177004 (12h48m25.709279581s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:51.194180164 (12h48m25.709282731s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.254375304 (12h48m25.769477873s) [upload=13178]: Done uploading segments... Aug 7 22:27:51.254404389 (12h48m25.76950696s) [upload=13178]: Upload finished. err: <nil> Aug 7 22:28:44.417417956 (12h49m18.932520542s) [upload=13188]: Starting upload Aug 7 22:28:44.417530389 (12h49m18.932632976s) [upload=13188]: Done starting upload Aug 7 22:28:44.417616643 (12h49m18.93271922s) [upload=13188]: Uploading segments... Aug 7 22:28:44.417757556 (12h49m18.932860142s) [upload=13188]: Got next segment. Inline: true Aug 7 22:28:44.417764563 (12h49m18.932867158s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:28:44.417818209 (12h49m18.932920789s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:44.417825102 (12h49m18.932927678s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:44.417832015 (12h49m18.932934599s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:37.965879533 (12h50m12.480982114s) [upload=13188]: Done uploading segments... Aug 7 22:29:37.965901992 (12h50m12.481004571s) [upload=13188]: Upload finished. err: <nil> Duration: 229.27509ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..XX................................] duration: 49.053646ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:46:47 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 49.078625ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..........................XXXXXXXXXXX.....................] duration: 657.028957ms FAIL (2025-08-07 22:46:46 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13121]: Uploading piece... Aug 7 22:23:58.634335933 (12h44m33.149438515s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634434236 (12h44m33.149536818s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634966805 (12h44m33.150069389s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634978506 (12h44m33.150081095s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635474598 (12h44m33.150577196s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635575111 (12h44m33.150677682s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635938157 (12h44m33.151040757s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.636367049 (12h44m33.151469627s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.637426749 (12h44m33.152529321s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638866353 (12h44m33.153968932s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638962747 (12h44m33.154065344s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.640916525 (12h44m33.156019112s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642334610 (12h44m33.15743719s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642802034 (12h44m33.157904611s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.644381227 (12h44m33.159483821s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.645186821 (12h44m33.160289426s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.646806610 (12h44m33.161909187s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.722673641 (12h44m33.237776211s) [upload=13121]: Next returned nil segment Aug 7 22:23:58.722691706 (12h44m33.237794289s) [upload=13121]: Waiting for error group managing segments... Aug 7 22:23:58.738782489 (12h44m33.253885056s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.741762463 (12h44m33.25686505s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.744959821 (12h44m33.260062396s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.751545661 (12h44m33.266648241s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.762632925 (12h44m33.277735495s) [noise=true , address=45.91.116.13:4168 , piece_id=5VCOHS7OBO4V2J4FHEQOMMWDK6FCNO7DTOPIURWKUG6XJIZJM2CQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.768246667 (12h44m33.28334923s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770018294 (12h44m33.285120863s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770550717 (12h44m33.285653287s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.772395609 (12h44m33.287498175s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.781308530 (12h44m33.296411093s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.782791410 (12h44m33.297893995s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.875949099 (12h44m33.391051663s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.108952090 (12h44m33.624054668s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158185576 (12h44m33.673288146s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158207517 (12h44m33.673310081s) [seg_pos={0 0}, upload=13121]: Segment reached optimal threshold of 14 pieces. Aug 7 22:23:59.158643218 (12h44m33.6737458s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CiiMtdAoMhPZwSqwraaQcrrx4vZw6L74vadw712VQbRLpNzKN): protocol: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection Aug 7 22:23:59.158730175 (12h44m33.673832737s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection Aug 7 22:23:59.158793697 (12h44m33.673896263s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection Aug 7 22:23:59.158849477 (12h44m33.673952045s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection Aug 7 22:23:59.158861716 (12h44m33.673964283s) [seg_pos={0 0}, upload=13121]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:23:59.158927631 (12h44m33.674030209s) [upload=13121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:23:59.158932133 (12h44m33.6740347s) [upload=13121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:59.182095456 (12h50m33.697198026s) [upload=13121]: Done uploading segments... Aug 7 22:29:59.182119282 (12h50m33.69722185s) [upload=13121]: Upload finished. err: metaclient: context deadline exceeded Duration: 657.056973ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..XX.........................X.XX..........................] duration: 515.845428ms FAIL (2025-08-07 22:46:41 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=12957]: Uploading segments... Aug 7 22:12:11.442470431 (12h32m45.957573001s) [upload=12957]: Got next segment. Inline: false Aug 7 22:12:11.442536399 (12h32m45.957638969s) [upload=12957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.442542497 (12h32m45.957645072s) [upload=12957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:41.409489142 (12h33m15.924591718s) [upload=12957]: Next segment err: context deadline exceeded Aug 7 22:12:41.409585580 (12h33m15.924688156s) [upload=12957]: Done uploading segments... Aug 7 22:12:41.409635672 (12h33m15.92473824s) [upload=12957]: Upload finished. err: context deadline exceeded Aug 7 22:12:41.517765572 (12h33m16.032868143s) [upload=12968]: Starting upload Aug 7 22:12:41.517838005 (12h33m16.032940574s) [upload=12968]: Done starting upload Aug 7 22:12:41.517963005 (12h33m16.033065573s) [upload=12968]: Uploading segments... Aug 7 22:12:41.518335606 (12h33m16.033438184s) [upload=12968]: Got next segment. Inline: false Aug 7 22:12:41.518426941 (12h33m16.033529524s) [upload=12968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:41.518434836 (12h33m16.03353742s) [upload=12968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:48.560277241 (12h33m23.075379809s) [seg_pos={0 0}, upload=12968]: Begin upload segment... Aug 7 22:12:48.560321663 (12h33m23.075424233s) [seg_pos={0 0}, upload=12968]: Done begin upload segment. Aug 7 22:12:48.560346254 (12h33m23.075448833s) [noise=true , address=77.90.31.21:4878 , piece_id=AYEE7NAJIYWU72ZKPBR44ZRU7T7HUB4FPAHH6V3KYKBSULV3AJMQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.560385235 (12h33m23.075487817s) [noise=true , address=104.36.82.4:4802 , piece_id=WUKRWI3KJKXS2KBP5WRMPI2UGYT7RKQ6KBBOAPPW6Z5ISN72VIWQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.560895927 (12h33m23.075998514s) [noise=true , address=45.91.116.5:4297 , piece_id=5TO5AKA4MGTPNPH6VWPA4GIDECUKKPFWSR5VZ3263TYANZ4BZGPA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.561270415 (12h33m23.076372997s) [noise=true , address=45.91.116.9:4262 , piece_id=UGR7YOFEX6MXTFNZPJ7EKO5DH3V7R47FOKI5ZNAHB4ZYH7J7UHKA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.562228406 (12h33m23.077330973s) [noise=true , address=45.91.116.11:4117 , piece_id=PQYC55OVQBT5EFFWHBMXXDFBGCDNBZFAYJR42IK5JEDDAXXWDJYA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.563480004 (12h33m23.078582611s) [noise=true , address=45.91.116.14:4190 , piece_id=3U7NYGD2JPOBZAHNJ2GUNUA3GE6JYRLJBMYQTEREUVWBR6QWAM6A, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.563854062 (12h33m23.078956648s) [noise=true , address=45.91.116.11:4118 , piece_id=A5SGTLAHM6E3N7AHB7U3KGZFYHDKO4JBE2IHNH76YKJUR54TGMZA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.564192678 (12h33m23.079295261s) [noise=true , address=104.36.82.12:4151 , piece_id=BV2GWAIQGUQ3BKVWHDBQ2RK772VQTLGYJ4WZU5VPALAGO7VOAZMA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.564542455 (12h33m23.079645037s) [noise=true , address=77.90.31.17:4771 , piece_id=OZMRVKETNTDWOLXUFP3IJLGQENMWSWB3MOTJKXXRUEDQHOFV6LQA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565038234 (12h33m23.080140824s) [noise=true , address=77.90.31.16:4733 , piece_id=LB4CQZC5X7QUEYL5OAQ4A4MA6OFO6LPC7RKCLN5AT6V6UGFHAKEQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565530690 (12h33m23.08063327s) [noise=true , address=77.90.31.10:4520 , piece_id=BKGMZCYXKDUXT42DSCXXLBKMYASXQWCG43HN4NJT76A5EESUNI3Q, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565683172 (12h33m23.080785758s) [noise=true , address=45.91.116.6:4975 , piece_id=KALV4QK4ML56PQUNJFDQXHTXZ5VOJYWZMXQNGHA6U5FLBXEW46TA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566145847 (12h33m23.081248429s) [noise=true , address=77.90.31.11:4553 , piece_id=TY7MWYZNDLY7CMDYALZFEYN2HE64TUEYW522DJKBFL463MMQIPOA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566452821 (12h33m23.0815554s) [noise=true , address=77.90.31.18:4805 , piece_id=NP4IZUJHXPID26UBGUMTHYRM2TEDF2BHPAE5Z4B34OVHJ4RRX5LA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566765755 (12h33m23.081868318s) [noise=true , address=77.90.31.21:4877 , piece_id=53XKORZS4TOBNGN2VM7DF55MS4NCZH7NX7F4ZZHDMJCKNT3YPVZQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.567074238 (12h33m23.082176801s) [noise=true , address=104.36.82.6:4876 , piece_id=MFX6SCYKRA4PAZ4VJ3JMKK6THX5MOCE4SBENR6NXDLOWA7C2GGJQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.674430176 (12h33m23.189532754s) [upload=12968]: Next returned nil segment Aug 7 22:12:48.674443729 (12h33m23.189546305s) [upload=12968]: Waiting for error group managing segments... Aug 7 22:12:48.677024873 (12h33m23.192127438s) [noise=true , address=77.90.31.11:4553 , piece_id=TY7MWYZNDLY7CMDYALZFEYN2HE64TUEYW522DJKBFL463MMQIPOA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.677316512 (12h33m23.192419097s) [noise=true , address=45.91.116.5:4297 , piece_id=5TO5AKA4MGTPNPH6VWPA4GIDECUKKPFWSR5VZ3263TYANZ4BZGPA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.677774329 (12h33m23.192876899s) [noise=true , address=45.91.116.9:4262 , piece_id=UGR7YOFEX6MXTFNZPJ7EKO5DH3V7R47FOKI5ZNAHB4ZYH7J7UHKA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678157641 (12h33m23.193260219s) [noise=true , address=45.91.116.11:4118 , piece_id=A5SGTLAHM6E3N7AHB7U3KGZFYHDKO4JBE2IHNH76YKJUR54TGMZA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678319308 (12h33m23.193421874s) [noise=true , address=45.91.116.11:4117 , piece_id=PQYC55OVQBT5EFFWHBMXXDFBGCDNBZFAYJR42IK5JEDDAXXWDJYA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678828950 (12h33m23.193931524s) [noise=true , address=45.91.116.6:4975 , piece_id=KALV4QK4ML56PQUNJFDQXHTXZ5VOJYWZMXQNGHA6U5FLBXEW46TA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678921127 (12h33m23.194023707s) [noise=true , address=45.91.116.14:4190 , piece_id=3U7NYGD2JPOBZAHNJ2GUNUA3GE6JYRLJBMYQTEREUVWBR6QWAM6A, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.679383015 (12h33m23.194485578s) [noise=true , address=77.90.31.21:4877 , piece_id=53XKORZS4TOBNGN2VM7DF55MS4NCZH7NX7F4ZZHDMJCKNT3YPVZQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.679808178 (12h33m23.194910761s) [noise=true , address=77.90.31.18:4805 , piece_id=NP4IZUJHXPID26UBGUMTHYRM2TEDF2BHPAE5Z4B34OVHJ4RRX5LA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.680957829 (12h33m23.196060398s) [noise=true , address=77.90.31.21:4878 , piece_id=AYEE7NAJIYWU72ZKPBR44ZRU7T7HUB4FPAHH6V3KYKBSULV3AJMQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.681390098 (12h33m23.196492675s) [noise=true , address=104.36.82.4:4802 , piece_id=WUKRWI3KJKXS2KBP5WRMPI2UGYT7RKQ6KBBOAPPW6Z5ISN72VIWQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.681868182 (12h33m23.19697076s) [noise=true , address=77.90.31.16:4733 , piece_id=LB4CQZC5X7QUEYL5OAQ4A4MA6OFO6LPC7RKCLN5AT6V6UGFHAKEQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.682434566 (12h33m23.197537204s) [noise=true , address=77.90.31.17:4771 , piece_id=OZMRVKETNTDWOLXUFP3IJLGQENMWSWB3MOTJKXXRUEDQHOFV6LQA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.682445484 (12h33m23.197548047s) [seg_pos={0 0}, upload=12968]: Segment reached optimal threshold of 13 pieces. Aug 7 22:12:48.682659606 (12h33m23.197762177s) [noise=true , address=77.90.31.10:4520 , piece_id=BKGMZCYXKDUXT42DSCXXLBKMYASXQWCG43HN4NJT76A5EESUNI3Q, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:48.682743670 (12h33m23.197846239s) [noise=true , address=104.36.82.12:4151 , piece_id=BV2GWAIQGUQ3BKVWHDBQ2RK772VQTLGYJ4WZU5VPALAGO7VOAZMA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection; write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection Aug 7 22:12:48.682881386 (12h33m23.197983962s) [noise=true , address=104.36.82.6:4876 , piece_id=MFX6SCYKRA4PAZ4VJ3JMKK6THX5MOCE4SBENR6NXDLOWA7C2GGJQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: read tcp 159.223.143.39:35642->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 22:12:48.682925198 (12h33m23.198027778s) [seg_pos={0 0}, upload=12968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:12:48.683071518 (12h33m23.198174098s) [upload=12968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:48.683078012 (12h33m23.19818059s) [upload=12968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:26.914480867 (12h50m1.429583435s) [upload=12968]: Done uploading segments... Aug 7 22:29:26.914503234 (12h50m1.42960581s) [upload=12968]: Upload finished. err: metaclient: context deadline exceeded Duration: 515.875535ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXX...X..........................] duration: 514.956638ms FAIL (2025-08-07 22:46:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.988892ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXXX.XX.X......................] duration: 318.263238ms FAIL (2025-08-07 22:46:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 318.301295ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXX..X.X...........................] duration: 271.414054ms FAIL (2025-08-07 22:46:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 271.442816ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [................XXXXXXXXXXX.XXXXXX..........................] duration: 734.428632ms FAIL (2025-08-07 22:46:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 734.45572ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXXX.X..........................] duration: 246.084137ms FAIL (2025-08-07 22:46:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g 7 22:20:59.026413148 (12h41m33.541515725s) [upload=13081]: Uploading segments... Aug 7 22:20:59.026487957 (12h41m33.541590544s) [upload=13081]: Got next segment. Inline: true Aug 7 22:20:59.026494660 (12h41m33.54159724s) [upload=13081]: Waiting for error group managing segments... Aug 7 22:20:59.026548283 (12h41m33.541650867s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.026554014 (12h41m33.541656591s) [upload=13081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.026557057 (12h41m33.541659621s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:28.995870666 (12h42m3.510973249s) [upload=13081]: Done uploading segments... Aug 7 22:21:28.995890051 (12h42m3.510992681s) [upload=13081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:29.129769938 (12h42m3.644872515s) [upload=13090]: Starting upload Aug 7 22:21:29.129856429 (12h42m3.644959013s) [upload=13090]: Done starting upload Aug 7 22:21:29.129943698 (12h42m3.645046276s) [upload=13090]: Uploading segments... Aug 7 22:21:29.129985259 (12h42m3.645087845s) [upload=13090]: Got next segment. Inline: true Aug 7 22:21:29.129990476 (12h42m3.645093052s) [upload=13090]: Waiting for error group managing segments... Aug 7 22:21:29.130027566 (12h42m3.645130145s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:29.130031892 (12h42m3.645134469s) [upload=13090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:29.130035507 (12h42m3.645138082s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:59.086555047 (12h42m33.601657631s) [upload=13090]: Done uploading segments... Aug 7 22:21:59.086578788 (12h42m33.601681377s) [upload=13090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:21.787894978 (12h42m56.302997579s) [upload=13102]: Starting upload Aug 7 22:22:21.787975806 (12h42m56.303078386s) [upload=13102]: Done starting upload Aug 7 22:22:21.788049720 (12h42m56.303152297s) [upload=13102]: Uploading segments... Aug 7 22:22:21.788092210 (12h42m56.303194795s) [upload=13102]: Got next segment. Inline: true Aug 7 22:22:21.788097559 (12h42m56.303200126s) [upload=13102]: Waiting for error group managing segments... Aug 7 22:22:21.788141372 (12h42m56.303243949s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:21.788146123 (12h42m56.30324876s) [upload=13102]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:21.788149819 (12h42m56.303252395s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:51.758466888 (12h43m26.273569473s) [upload=13102]: Done uploading segments... Aug 7 22:22:51.758489416 (12h43m26.273591984s) [upload=13102]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:17.561525115 (12h43m52.076627681s) [upload=13117]: Starting upload Aug 7 22:23:17.561618947 (12h43m52.076721524s) [upload=13117]: Done starting upload Aug 7 22:23:17.561741793 (12h43m52.07684436s) [upload=13117]: Uploading segments... Aug 7 22:23:17.561843225 (12h43m52.076945802s) [upload=13117]: Got next segment. Inline: true Aug 7 22:23:17.561848606 (12h43m52.076951175s) [upload=13117]: Waiting for error group managing segments... Aug 7 22:23:17.561909318 (12h43m52.077011885s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:17.561914564 (12h43m52.077017202s) [upload=13117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:17.561918167 (12h43m52.077020738s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:47.532068483 (12h44m22.047171052s) [upload=13117]: Done uploading segments... Aug 7 22:23:47.532095698 (12h44m22.047198268s) [upload=13117]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:55.295508312 (12h44m29.810610893s) [upload=13123]: Starting upload Aug 7 22:23:55.295607041 (12h44m29.81070962s) [upload=13123]: Done starting upload Aug 7 22:23:55.295735905 (12h44m29.810838474s) [upload=13123]: Uploading segments... Aug 7 22:23:55.295799978 (12h44m29.810902554s) [upload=13123]: Got next segment. Inline: true Aug 7 22:23:55.295805637 (12h44m29.810908214s) [upload=13123]: Waiting for error group managing segments... Aug 7 22:23:55.295849955 (12h44m29.810952532s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:55.295854812 (12h44m29.810957387s) [upload=13123]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:55.295858141 (12h44m29.810960717s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:25.257624495 (12h44m59.772727074s) [upload=13123]: Done uploading segments... Aug 7 22:24:25.257647063 (12h44m59.77274964s) [upload=13123]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:51.263752764 (12h45m25.778855407s) [upload=13135]: Starting upload Aug 7 22:24:51.263877660 (12h45m25.778980247s) [upload=13135]: Done starting upload Aug 7 22:24:51.263966776 (12h45m25.779069345s) [upload=13135]: Uploading segments... Aug 7 22:24:51.264018780 (12h45m25.779121365s) [upload=13135]: Got next segment. Inline: true Aug 7 22:24:51.264023895 (12h45m25.779126462s) [upload=13135]: Waiting for error group managing segments... Aug 7 22:24:51.264067265 (12h45m25.779169838s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:51.264097890 (12h45m25.779200467s) [upload=13135]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:51.264101967 (12h45m25.779204534s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:21.220725953 (12h45m55.735828536s) [upload=13135]: Done uploading segments... Aug 7 22:25:21.220748855 (12h45m55.735851436s) [upload=13135]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:25.019445687 (12h45m59.534548303s) [upload=13145]: Starting upload Aug 7 22:25:25.019581109 (12h45m59.534683692s) [upload=13145]: Done starting upload Aug 7 22:25:25.019703288 (12h45m59.534805856s) [upload=13145]: Uploading segments... Aug 7 22:25:25.019823488 (12h45m59.534926072s) [upload=13145]: Got next segment. Inline: true Aug 7 22:25:25.019831359 (12h45m59.534933942s) [upload=13145]: Waiting for error group managing segments... Aug 7 22:25:25.019906993 (12h45m59.535009557s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:25.019929793 (12h45m59.535032371s) [upload=13145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:25.019934887 (12h45m59.535037461s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:54.989474261 (12h46m29.504576827s) [upload=13145]: Done uploading segments... Aug 7 22:25:54.989504165 (12h46m29.50460673s) [upload=13145]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:39.497160053 (12h47m14.012262635s) [upload=13160]: Starting upload Aug 7 22:26:39.497281249 (12h47m14.01238383s) [upload=13160]: Done starting upload Aug 7 22:26:39.497374185 (12h47m14.012476751s) [upload=13160]: Uploading segments... Aug 7 22:26:39.497465038 (12h47m14.012567626s) [upload=13160]: Got next segment. Inline: true Aug 7 22:26:39.497472429 (12h47m14.012575009s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:26:39.497544385 (12h47m14.012646971s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:39.497550182 (12h47m14.012652768s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:39.497554536 (12h47m14.012657133s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.465616489 (12h47m43.980719066s) [upload=13160]: Done uploading segments... Aug 7 22:27:09.465659032 (12h47m43.980761614s) [upload=13160]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:09.584230741 (12h47m44.099333328s) [upload=13164]: Starting upload Aug 7 22:27:09.584314650 (12h47m44.099417234s) [upload=13164]: Done starting upload Aug 7 22:27:09.584433539 (12h47m44.099536113s) [upload=13164]: Uploading segments... Aug 7 22:27:09.584547339 (12h47m44.099649917s) [upload=13164]: Got next segment. Inline: true Aug 7 22:27:09.584576624 (12h47m44.09967921s) [upload=13164]: Waiting for error group managing segments... Aug 7 22:27:09.584764211 (12h47m44.099866798s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:09.584774497 (12h47m44.099877084s) [upload=13164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:09.584779600 (12h47m44.099882182s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.656944239 (12h47m44.172046803s) [upload=13164]: Done uploading segments... Aug 7 22:27:09.656961099 (12h47m44.172063662s) [upload=13164]: Upload finished. err: <nil> Aug 7 22:28:22.451723369 (12h48m56.966825932s) [upload=13182]: Starting upload Aug 7 22:28:22.451781104 (12h48m56.96688368s) [upload=13182]: Done starting upload Aug 7 22:28:22.451842252 (12h48m56.966944827s) [upload=13182]: Uploading segments... Aug 7 22:28:22.451894112 (12h48m56.9669967s) [upload=13182]: Got next segment. Inline: true Aug 7 22:28:22.451901034 (12h48m56.967003616s) [upload=13182]: Waiting for error group managing segments... Aug 7 22:28:22.451944887 (12h48m56.967047449s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:22.451948845 (12h48m56.967051407s) [upload=13182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:22.451951261 (12h48m56.967053837s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:52.423300054 (12h49m26.938402631s) [upload=13182]: Done uploading segments... Aug 7 22:28:52.423324837 (12h49m26.938427414s) [upload=13182]: Upload finished. err: metaclient: context deadline exceeded Duration: 246.11165ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXXXX...X.........................] duration: 621.682132ms FAIL (2025-08-07 22:46:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 621.724926ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [..................XXXXXXXXXXX.X..X.X........................] duration: 245.967666ms FAIL (2025-08-07 22:46:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:45.259131358 (12h41m19.774233925s) [upload=13078]: Starting upload Aug 7 22:20:45.259192437 (12h41m19.774295013s) [upload=13078]: Done starting upload Aug 7 22:20:45.259251322 (12h41m19.774353897s) [upload=13078]: Uploading segments... Aug 7 22:20:45.259289829 (12h41m19.774392414s) [upload=13078]: Got next segment. Inline: true Aug 7 22:20:45.259294478 (12h41m19.774397054s) [upload=13078]: Waiting for error group managing segments... Aug 7 22:20:45.259332525 (12h41m19.774435101s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:45.259337069 (12h41m19.774439635s) [upload=13078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:45.259340263 (12h41m19.77444284s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:45.229308083 (12h42m19.744410669s) [upload=13078]: Done uploading segments... Aug 7 22:21:45.229380638 (12h42m19.744483223s) [upload=13078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:59.915154556 (12h42m34.430257139s) [upload=13097]: Starting upload Aug 7 22:21:59.915242531 (12h42m34.430345108s) [upload=13097]: Done starting upload Aug 7 22:21:59.915358515 (12h42m34.430461092s) [upload=13097]: Uploading segments... Aug 7 22:21:59.915398067 (12h42m34.430500653s) [upload=13097]: Got next segment. Inline: true Aug 7 22:21:59.915403151 (12h42m34.430505727s) [upload=13097]: Waiting for error group managing segments... Aug 7 22:21:59.915458296 (12h42m34.430560863s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:59.915463422 (12h42m34.430565998s) [upload=13097]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:59.915466628 (12h42m34.430569269s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.883702618 (12h43m34.398805207s) [upload=13097]: Done uploading segments... Aug 7 22:22:59.883744759 (12h43m34.398847337s) [upload=13097]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:00.217702003 (12h43m34.732804585s) [upload=13110]: Starting upload Aug 7 22:23:00.217805899 (12h43m34.732908483s) [upload=13110]: Done starting upload Aug 7 22:23:00.217896494 (12h43m34.732999072s) [upload=13110]: Uploading segments... Aug 7 22:23:00.217970831 (12h43m34.733073418s) [upload=13110]: Got next segment. Inline: true Aug 7 22:23:00.217979414 (12h43m34.733081991s) [upload=13110]: Waiting for error group managing segments... Aug 7 22:23:00.218043817 (12h43m34.733146394s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:00.218051159 (12h43m34.733153734s) [upload=13110]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:00.218054412 (12h43m34.733156988s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:00.464344185 (12h43m34.979446778s) [upload=13110]: Done uploading segments... Aug 7 22:23:00.464376970 (12h43m34.979479551s) [upload=13110]: Upload finished. err: <nil> Aug 7 22:24:07.143696312 (12h44m41.658798899s) [upload=13125]: Starting upload Aug 7 22:24:07.143811006 (12h44m41.658913589s) [upload=13125]: Done starting upload Aug 7 22:24:07.143914011 (12h44m41.659016577s) [upload=13125]: Uploading segments... Aug 7 22:24:07.143973667 (12h44m41.659076252s) [upload=13125]: Got next segment. Inline: true Aug 7 22:24:07.143979447 (12h44m41.659082034s) [upload=13125]: Waiting for error group managing segments... Aug 7 22:24:07.144045718 (12h44m41.659148297s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:07.144051783 (12h44m41.659154361s) [upload=13125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:07.144057848 (12h44m41.659160429s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.110905816 (12h45m41.626008396s) [upload=13125]: Done uploading segments... Aug 7 22:25:07.110929433 (12h45m41.626032014s) [upload=13125]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:07.223172827 (12h45m41.738275404s) [upload=13138]: Starting upload Aug 7 22:25:07.223254341 (12h45m41.738356918s) [upload=13138]: Done starting upload Aug 7 22:25:07.223368837 (12h45m41.738471414s) [upload=13138]: Uploading segments... Aug 7 22:25:07.223445919 (12h45m41.738548507s) [upload=13138]: Got next segment. Inline: true Aug 7 22:25:07.223452833 (12h45m41.738555412s) [upload=13138]: Waiting for error group managing segments... Aug 7 22:25:07.223511692 (12h45m41.738614278s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:07.223518073 (12h45m41.738620658s) [upload=13138]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:07.223521493 (12h45m41.738624078s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.315623341 (12h45m41.830725924s) [upload=13138]: Done uploading segments... Aug 7 22:25:07.315640785 (12h45m41.830743361s) [upload=13138]: Upload finished. err: <nil> Aug 7 22:26:03.367436251 (12h46m37.882538842s) [upload=13151]: Starting upload Aug 7 22:26:03.367515051 (12h46m37.882617639s) [upload=13151]: Done starting upload Aug 7 22:26:03.367596048 (12h46m37.882698629s) [upload=13151]: Uploading segments... Aug 7 22:26:03.367644909 (12h46m37.8827475s) [upload=13151]: Got next segment. Inline: true Aug 7 22:26:03.367650467 (12h46m37.882753051s) [upload=13151]: Waiting for error group managing segments... Aug 7 22:26:03.367698612 (12h46m37.882801214s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:03.367705011 (12h46m37.882807591s) [upload=13151]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:03.367709618 (12h46m37.882812202s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:03.460311519 (12h46m37.975414085s) [upload=13151]: Done uploading segments... Aug 7 22:26:03.460350352 (12h46m37.975452923s) [upload=13151]: Upload finished. err: <nil> Aug 7 22:26:51.117138988 (12h47m25.632241558s) [upload=13162]: Starting upload Aug 7 22:26:51.117235608 (12h47m25.632338187s) [upload=13162]: Done starting upload Aug 7 22:26:51.117347075 (12h47m25.632449638s) [upload=13162]: Uploading segments... Aug 7 22:26:51.117436995 (12h47m25.632539563s) [upload=13162]: Got next segment. Inline: true Aug 7 22:26:51.117443130 (12h47m25.632545707s) [upload=13162]: Waiting for error group managing segments... Aug 7 22:26:51.117504554 (12h47m25.632607133s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:51.117509417 (12h47m25.632611996s) [upload=13162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:51.117512811 (12h47m25.63261539s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.086979010 (12h48m25.602081586s) [upload=13162]: Done uploading segments... Aug 7 22:27:51.086998672 (12h48m25.602101249s) [upload=13162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:51.193960256 (12h48m25.709062823s) [upload=13178]: Starting upload Aug 7 22:27:51.194024831 (12h48m25.709127407s) [upload=13178]: Done starting upload Aug 7 22:27:51.194084091 (12h48m25.709186657s) [upload=13178]: Uploading segments... Aug 7 22:27:51.194125626 (12h48m25.709228211s) [upload=13178]: Got next segment. Inline: true Aug 7 22:27:51.194132041 (12h48m25.709234607s) [upload=13178]: Waiting for error group managing segments... Aug 7 22:27:51.194172625 (12h48m25.709275192s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:51.194177004 (12h48m25.709279581s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:51.194180164 (12h48m25.709282731s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.254375304 (12h48m25.769477873s) [upload=13178]: Done uploading segments... Aug 7 22:27:51.254404389 (12h48m25.76950696s) [upload=13178]: Upload finished. err: <nil> Aug 7 22:28:44.417417956 (12h49m18.932520542s) [upload=13188]: Starting upload Aug 7 22:28:44.417530389 (12h49m18.932632976s) [upload=13188]: Done starting upload Aug 7 22:28:44.417616643 (12h49m18.93271922s) [upload=13188]: Uploading segments... Aug 7 22:28:44.417757556 (12h49m18.932860142s) [upload=13188]: Got next segment. Inline: true Aug 7 22:28:44.417764563 (12h49m18.932867158s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:28:44.417818209 (12h49m18.932920789s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:44.417825102 (12h49m18.932927678s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:44.417832015 (12h49m18.932934599s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:37.965879533 (12h50m12.480982114s) [upload=13188]: Done uploading segments... Aug 7 22:29:37.965901992 (12h50m12.481004571s) [upload=13188]: Upload finished. err: <nil> Duration: 245.997746ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................XXXXXXXXX.X.........................] duration: 1.834587968s FAIL (2025-08-07 22:46:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.834615794s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.................XXXXXXXXXXX.XXXXXX.........................] duration: 782.519086ms FAIL (2025-08-07 22:46:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 782.547234ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........................X................] duration: 622.536558ms FAIL (2025-08-07 22:46:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== g_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214654258 (12h54m14.729756841s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.214886232 (12h54m14.729988875s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215184221 (12h54m14.730286807s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215580311 (12h54m14.730682897s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.215600089 (12h54m14.730702668s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216063638 (12h54m14.731166221s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216221465 (12h54m14.73132405s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216499611 (12h54m14.73160219s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216784173 (12h54m14.731886753s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.216959396 (12h54m14.732061963s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217229150 (12h54m14.732331732s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217364444 (12h54m14.732467028s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.217574188 (12h54m14.732676766s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.218700401 (12h54m14.733802981s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.219113148 (12h54m14.734215718s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Uploading piece... Aug 7 22:33:40.307711467 (12h54m14.82281403s) [upload=13273]: Next returned nil segment Aug 7 22:33:40.307743585 (12h54m14.822846164s) [upload=13273]: Waiting for error group managing segments... Aug 7 22:33:40.321114866 (12h54m14.836217441s) [noise=true , address=45.91.116.9:4294 , piece_id=UMCVXJVJ7VWVLOXYVN4KSE5VHM2YXJWOD45364KGN2T62PGBYNOA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.321899105 (12h54m14.837001676s) [noise=true , address=45.91.116.4:4943 , piece_id=4U7YPCLZI7HIZGGBGVWIWJLBRNPGIYOVVUD2UEJJVCMZMC426JIA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.330294085 (12h54m14.845396652s) [noise=true , address=79.127.219.51:4181 , piece_id=6JHBVXMBPJZUANM7XYIWUU5546ZWOQ2Y5G4DNJEUSSOFQEQ4CYWQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.331982437 (12h54m14.847085014s) [noise=true , address=104.36.82.61:4384 , piece_id=BKB3NUVUHGPCCSFSL67TQXJPPPCV6G4OTEZXQGN3ODOKFKLLZOUQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.334537739 (12h54m14.849640313s) [noise=true , address=104.36.82.54:4176 , piece_id=BXPHUWUTUPP3TH2LLOPO6SM43MYUCWJ7VRPKIHV5RF7Z63KPGR4Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.335955579 (12h54m14.851058147s) [noise=true , address=104.36.82.47:4537 , piece_id=M7UXNTM4OYDMXOSBH723A64FWQV4WR6ZXCJ2VT427E6OWARGR7AQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.340231956 (12h54m14.855334526s) [noise=true , address=77.90.31.11:4559 , piece_id=JSL5C3XIXINYAVAVTQ7J3QD3WWRG6FKP4H4DGJLDOPWCGI7CNAFA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.349513208 (12h54m14.864615777s) [noise=true , address=45.91.116.7:4039 , piece_id=AMJ45JG2ESL4UYNXPJZPCKGOR5WA57EYLNQHEEA7SY5KC762EREQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.358627538 (12h54m14.873730105s) [noise=true , address=104.36.82.47:4518 , piece_id=BLKMQ2NJY3DWDNTSTHIIZ2OVNXZ6TOEWZOWHBX6N5YZOQDFKL4TQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.436339846 (12h54m14.951442414s) [noise=true , address=104.36.82.55:4193 , piece_id=IYZ4ERNRWHEICGVPP3HZ7FA45JMV7JOSSFSSUQ632ENB4YS6LWTA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.543248537 (12h54m15.058351106s) [noise=true , address=104.36.82.52:4094 , piece_id=RXR2PS2E2GDDJJAKZ2KAHCYIT5SF5JBHPJWE57J5P5UASJYZM6EQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.650678621 (12h54m15.165781209s) [noise=true , address=104.36.82.8:4980 , piece_id=D5YC5QADQV6E6MRLCC3UOKKORO56B7BGPP23GK25QP2HNXDJHC5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.786445525 (12h54m15.301548105s) [noise=true , address=104.36.82.18:4399 , piece_id=ZISIDERXJ7FWYKXPDNASORAJCQWDJF7B7XTTKGBAMB3MN5DPGC6Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907165087 (12h54m15.422267656s) [noise=true , address=64.49.13.137:4501 , piece_id=U2ZWPC6ESWKOHUTQ7ACAB7X4FHBNANOKOPONSI4WMEO4BCNAYE5Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: <nil> Aug 7 22:33:40.907202527 (12h54m15.422305104s) [seg_pos={0 0}, upload=13273]: Segment reached optimal threshold of 14 pieces. Aug 7 22:33:40.907372225 (12h54m15.422474801s) [noise=true , address=104.36.82.62:4436 , piece_id=N76AMFHYCYEZQAONCUKTKZFYLHNZK6OHAZSZECQVZXJWUZVEIHQQ, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QrrGrMHYAjyYiTdTXDUy2uH1j2p1qtUVJaDtais2GowJqBPPu): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.907642442 (12h54m15.422745018s) [noise=true , address=104.36.82.38:4707 , piece_id=2JN7DEYY7HDLSSEDC63ERBT6XRROQ3QAH7H4JCMXIGPYJSK7P76Q, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1N7XaDABpimefP5xEz5Swk53fiZk4DTv5L8Hg5cNDgeG6gHP1T): protocol: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50010->104.36.82.38:4707: use of closed network connection Aug 7 22:33:40.907950578 (12h54m15.423053162s) [noise=true , address=104.36.82.60:4345 , piece_id=TGD52W5A5UJPG2BLDDF64UEIDANEGFWTLDPWVYOAQZAZ4XCUR2AA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59242->104.36.82.60:4345: use of closed network connection Aug 7 22:33:40.908057652 (12h54m15.423160235s) [noise=true , address=104.36.82.37:4681 , piece_id=REGVS57XVDDF4LAIVG4ACMAFUTLTBGZXOO7A64HIV4FAFFTVTWLA, seg_pos={0 0}, upload=13273]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YDoKEKcndBYDziWRgqHB1Ms2GnhKUG7qbj6qxZFrus8r1kXy2): protocol: expected piece hash; context canceled; EOF Aug 7 22:33:40.908080526 (12h54m15.423183103s) [seg_pos={0 0}, upload=13273]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:33:40.908212573 (12h54m15.423315155s) [upload=13273]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:33:40.908218832 (12h54m15.423321415s) [upload=13273]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:33:41.213299020 (12h54m15.728401607s) [upload=13273]: Done uploading segments... Aug 7 22:33:41.213326550 (12h54m15.728429128s) [upload=13273]: Upload finished. err: <nil> Aug 7 22:34:48.031458911 (12h55m22.546561477s) [upload=13295]: Starting upload Aug 7 22:34:48.031533867 (12h55m22.546636441s) [upload=13295]: Done starting upload Aug 7 22:34:48.031664339 (12h55m22.546766908s) [upload=13295]: Uploading segments... Aug 7 22:34:48.031895444 (12h55m22.546998028s) [upload=13295]: Got next segment. Inline: false Aug 7 22:34:48.031982233 (12h55m22.547084799s) [upload=13295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:34:48.031988106 (12h55m22.547090688s) [upload=13295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:34:48.040357077 (12h55m22.555459659s) [upload=13295]: Next segment err: metaclient: Too Many Requests Aug 7 22:34:48.040386485 (12h55m22.555489061s) [upload=13295]: Done uploading segments... Aug 7 22:34:48.040426787 (12h55m22.555529363s) [upload=13295]: Upload finished. err: metaclient: Too Many Requests Duration: 622.564685ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 115.514117ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:46:03 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 115.543752ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [........................X..XX...............................] duration: 41.383199ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:45:51 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 41.405979ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................XXXXXXXXX.X........................] duration: 1.832306313s FAIL (2025-08-07 22:45:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.832334217s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X..........................XXXXXXXXXXX....................] duration: 1.561355597s FAIL (2025-08-07 22:45:47 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13121]: Uploading piece... Aug 7 22:23:58.634335933 (12h44m33.149438515s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634434236 (12h44m33.149536818s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634966805 (12h44m33.150069389s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.634978506 (12h44m33.150081095s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635474598 (12h44m33.150577196s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635575111 (12h44m33.150677682s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.635938157 (12h44m33.151040757s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.636367049 (12h44m33.151469627s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.637426749 (12h44m33.152529321s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638866353 (12h44m33.153968932s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.638962747 (12h44m33.154065344s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.640916525 (12h44m33.156019112s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642334610 (12h44m33.15743719s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.642802034 (12h44m33.157904611s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.644381227 (12h44m33.159483821s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.645186821 (12h44m33.160289426s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.646806610 (12h44m33.161909187s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Uploading piece... Aug 7 22:23:58.722673641 (12h44m33.237776211s) [upload=13121]: Next returned nil segment Aug 7 22:23:58.722691706 (12h44m33.237794289s) [upload=13121]: Waiting for error group managing segments... Aug 7 22:23:58.738782489 (12h44m33.253885056s) [noise=true , address=45.91.116.5:4323 , piece_id=64LZJNIVNSXRBKP23QQA3VKOVHENZUXA7WAWIR6ZEQZCVVNSCI4Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.741762463 (12h44m33.25686505s) [noise=true , address=45.91.116.8:4051 , piece_id=5KEENOWYLJIOYUVEFEIZ3DLSMWXNKIX7QRGI3X6BFSUIWNGLEYYA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.744959821 (12h44m33.260062396s) [noise=true , address=104.36.82.36:4649 , piece_id=HERBYVKSIH54I2VKMILZAJIQNR42B7S4L4RVWUTBSS6FQJKAAZHA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.751545661 (12h44m33.266648241s) [noise=true , address=104.36.82.62:4421 , piece_id=JZQZN5WUFAAKUFTGRTG5INSZ5G2ZHGMU2WJFY6ILPZJDECMPVD7Q, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.762632925 (12h44m33.277735495s) [noise=true , address=45.91.116.13:4168 , piece_id=5VCOHS7OBO4V2J4FHEQOMMWDK6FCNO7DTOPIURWKUG6XJIZJM2CQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.768246667 (12h44m33.28334923s) [noise=true , address=104.36.82.22:4524 , piece_id=SNZ7OPH4ICJAG5QWZOXJTYC53NJ36JTZ4NLCO6QP5DW57EJKO7RA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770018294 (12h44m33.285120863s) [noise=true , address=104.36.82.35:4589 , piece_id=NPFDSCXS2E2ZDDKDTRWDFWUNOBIACVJ2ZPSHPDIR76PQWC4ME3RQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.770550717 (12h44m33.285653287s) [noise=true , address=104.36.82.6:4891 , piece_id=3HBW5VHTIPVAN6PWZOG6A4EZSS73KCJ2ZYBZW2J6B63N4FF5RWHQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.772395609 (12h44m33.287498175s) [noise=true , address=104.36.82.14:4245 , piece_id=4W6MMLAWGRFR64NXOPNSUR5BB3RWEWWI7V4QCRONK2RTMJNNKDZA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.781308530 (12h44m33.296411093s) [noise=true , address=104.36.82.9:4039 , piece_id=BBU7IWFW22DJBKWLEKKFCLITFVQ4TEKFG2L6MUHWAQ2JOKEYR7DQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.782791410 (12h44m33.297893995s) [noise=true , address=104.36.82.23:4461 , piece_id=BVQ3AMD6NJQ3RCVZTH4BT2PJLW4NASLDQ2DQQMF4IZNCGEAKER3A, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:58.875949099 (12h44m33.391051663s) [noise=true , address=104.36.82.8:4983 , piece_id=KOQFY7F2EV7MONNJKY3565M4CCVSIF3GBLJFWESCAEYFFA4ZIEXA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.108952090 (12h44m33.624054668s) [noise=true , address=104.36.82.37:4679 , piece_id=X23ZPXU7QJRB4O2M3SEECXB6FJJTWVBWORPJYVWPRDHOEH5XWHDA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158185576 (12h44m33.673288146s) [noise=true , address=104.36.82.5:4853 , piece_id=36S46MNGL7YR43445O6EP4XOF4K53KPNTKBXHFNMC2B54JHMKLNA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: <nil> Aug 7 22:23:59.158207517 (12h44m33.673310081s) [seg_pos={0 0}, upload=13121]: Segment reached optimal threshold of 14 pieces. Aug 7 22:23:59.158643218 (12h44m33.6737458s) [noise=true , address=64.49.13.138:4489 , piece_id=FQFNPC6RY5GG777WZQNGSZATEHDFYH7NX7HIW4AESDHWOYT3XNEA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CiiMtdAoMhPZwSqwraaQcrrx4vZw6L74vadw712VQbRLpNzKN): protocol: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56790->64.49.13.138:4489: use of closed network connection Aug 7 22:23:59.158730175 (12h44m33.673832737s) [noise=true , address=64.49.13.169:4706 , piece_id=H6Y5GDBHDHGZVXRMDPSJTTS7N5RJKN77ZW2E3WPEVZOLM6CIRMBQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53782->64.49.13.169:4706: use of closed network connection Aug 7 22:23:59.158793697 (12h44m33.673896263s) [noise=true , address=104.36.82.44:4872 , piece_id=4NINGCCX47U7L3V3HUAABNW3P4QQQWVN4QMUQXQ6HV2U7AV7JWMQ, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58856->104.36.82.44:4872: use of closed network connection Aug 7 22:23:59.158849477 (12h44m33.673952045s) [noise=true , address=64.49.13.27:4964 , piece_id=XU4SWAKH6HIOLZ2D5ESVBJ6SVKCGMXRHXOMJCIYBUGA676GHDDBA, seg_pos={0 0}, upload=13121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59794->64.49.13.27:4964: use of closed network connection Aug 7 22:23:59.158861716 (12h44m33.673964283s) [seg_pos={0 0}, upload=13121]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:23:59.158927631 (12h44m33.674030209s) [upload=13121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:23:59.158932133 (12h44m33.6740347s) [upload=13121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:59.182095456 (12h50m33.697198026s) [upload=13121]: Done uploading segments... Aug 7 22:29:59.182119282 (12h50m33.69722185s) [upload=13121]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.561383513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXX.X.........................] duration: 225.712343ms FAIL (2025-08-07 22:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g 7 22:20:59.026413148 (12h41m33.541515725s) [upload=13081]: Uploading segments... Aug 7 22:20:59.026487957 (12h41m33.541590544s) [upload=13081]: Got next segment. Inline: true Aug 7 22:20:59.026494660 (12h41m33.54159724s) [upload=13081]: Waiting for error group managing segments... Aug 7 22:20:59.026548283 (12h41m33.541650867s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:59.026554014 (12h41m33.541656591s) [upload=13081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:59.026557057 (12h41m33.541659621s) [upload=13081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:28.995870666 (12h42m3.510973249s) [upload=13081]: Done uploading segments... Aug 7 22:21:28.995890051 (12h42m3.510992681s) [upload=13081]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:29.129769938 (12h42m3.644872515s) [upload=13090]: Starting upload Aug 7 22:21:29.129856429 (12h42m3.644959013s) [upload=13090]: Done starting upload Aug 7 22:21:29.129943698 (12h42m3.645046276s) [upload=13090]: Uploading segments... Aug 7 22:21:29.129985259 (12h42m3.645087845s) [upload=13090]: Got next segment. Inline: true Aug 7 22:21:29.129990476 (12h42m3.645093052s) [upload=13090]: Waiting for error group managing segments... Aug 7 22:21:29.130027566 (12h42m3.645130145s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:29.130031892 (12h42m3.645134469s) [upload=13090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:29.130035507 (12h42m3.645138082s) [upload=13090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:59.086555047 (12h42m33.601657631s) [upload=13090]: Done uploading segments... Aug 7 22:21:59.086578788 (12h42m33.601681377s) [upload=13090]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:21.787894978 (12h42m56.302997579s) [upload=13102]: Starting upload Aug 7 22:22:21.787975806 (12h42m56.303078386s) [upload=13102]: Done starting upload Aug 7 22:22:21.788049720 (12h42m56.303152297s) [upload=13102]: Uploading segments... Aug 7 22:22:21.788092210 (12h42m56.303194795s) [upload=13102]: Got next segment. Inline: true Aug 7 22:22:21.788097559 (12h42m56.303200126s) [upload=13102]: Waiting for error group managing segments... Aug 7 22:22:21.788141372 (12h42m56.303243949s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:21.788146123 (12h42m56.30324876s) [upload=13102]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:21.788149819 (12h42m56.303252395s) [upload=13102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:51.758466888 (12h43m26.273569473s) [upload=13102]: Done uploading segments... Aug 7 22:22:51.758489416 (12h43m26.273591984s) [upload=13102]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:17.561525115 (12h43m52.076627681s) [upload=13117]: Starting upload Aug 7 22:23:17.561618947 (12h43m52.076721524s) [upload=13117]: Done starting upload Aug 7 22:23:17.561741793 (12h43m52.07684436s) [upload=13117]: Uploading segments... Aug 7 22:23:17.561843225 (12h43m52.076945802s) [upload=13117]: Got next segment. Inline: true Aug 7 22:23:17.561848606 (12h43m52.076951175s) [upload=13117]: Waiting for error group managing segments... Aug 7 22:23:17.561909318 (12h43m52.077011885s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:17.561914564 (12h43m52.077017202s) [upload=13117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:17.561918167 (12h43m52.077020738s) [upload=13117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:47.532068483 (12h44m22.047171052s) [upload=13117]: Done uploading segments... Aug 7 22:23:47.532095698 (12h44m22.047198268s) [upload=13117]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:55.295508312 (12h44m29.810610893s) [upload=13123]: Starting upload Aug 7 22:23:55.295607041 (12h44m29.81070962s) [upload=13123]: Done starting upload Aug 7 22:23:55.295735905 (12h44m29.810838474s) [upload=13123]: Uploading segments... Aug 7 22:23:55.295799978 (12h44m29.810902554s) [upload=13123]: Got next segment. Inline: true Aug 7 22:23:55.295805637 (12h44m29.810908214s) [upload=13123]: Waiting for error group managing segments... Aug 7 22:23:55.295849955 (12h44m29.810952532s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:55.295854812 (12h44m29.810957387s) [upload=13123]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:55.295858141 (12h44m29.810960717s) [upload=13123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:25.257624495 (12h44m59.772727074s) [upload=13123]: Done uploading segments... Aug 7 22:24:25.257647063 (12h44m59.77274964s) [upload=13123]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:51.263752764 (12h45m25.778855407s) [upload=13135]: Starting upload Aug 7 22:24:51.263877660 (12h45m25.778980247s) [upload=13135]: Done starting upload Aug 7 22:24:51.263966776 (12h45m25.779069345s) [upload=13135]: Uploading segments... Aug 7 22:24:51.264018780 (12h45m25.779121365s) [upload=13135]: Got next segment. Inline: true Aug 7 22:24:51.264023895 (12h45m25.779126462s) [upload=13135]: Waiting for error group managing segments... Aug 7 22:24:51.264067265 (12h45m25.779169838s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:51.264097890 (12h45m25.779200467s) [upload=13135]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:51.264101967 (12h45m25.779204534s) [upload=13135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:21.220725953 (12h45m55.735828536s) [upload=13135]: Done uploading segments... Aug 7 22:25:21.220748855 (12h45m55.735851436s) [upload=13135]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:25.019445687 (12h45m59.534548303s) [upload=13145]: Starting upload Aug 7 22:25:25.019581109 (12h45m59.534683692s) [upload=13145]: Done starting upload Aug 7 22:25:25.019703288 (12h45m59.534805856s) [upload=13145]: Uploading segments... Aug 7 22:25:25.019823488 (12h45m59.534926072s) [upload=13145]: Got next segment. Inline: true Aug 7 22:25:25.019831359 (12h45m59.534933942s) [upload=13145]: Waiting for error group managing segments... Aug 7 22:25:25.019906993 (12h45m59.535009557s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:25.019929793 (12h45m59.535032371s) [upload=13145]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:25.019934887 (12h45m59.535037461s) [upload=13145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:54.989474261 (12h46m29.504576827s) [upload=13145]: Done uploading segments... Aug 7 22:25:54.989504165 (12h46m29.50460673s) [upload=13145]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:39.497160053 (12h47m14.012262635s) [upload=13160]: Starting upload Aug 7 22:26:39.497281249 (12h47m14.01238383s) [upload=13160]: Done starting upload Aug 7 22:26:39.497374185 (12h47m14.012476751s) [upload=13160]: Uploading segments... Aug 7 22:26:39.497465038 (12h47m14.012567626s) [upload=13160]: Got next segment. Inline: true Aug 7 22:26:39.497472429 (12h47m14.012575009s) [upload=13160]: Waiting for error group managing segments... Aug 7 22:26:39.497544385 (12h47m14.012646971s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:39.497550182 (12h47m14.012652768s) [upload=13160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:39.497554536 (12h47m14.012657133s) [upload=13160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.465616489 (12h47m43.980719066s) [upload=13160]: Done uploading segments... Aug 7 22:27:09.465659032 (12h47m43.980761614s) [upload=13160]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:09.584230741 (12h47m44.099333328s) [upload=13164]: Starting upload Aug 7 22:27:09.584314650 (12h47m44.099417234s) [upload=13164]: Done starting upload Aug 7 22:27:09.584433539 (12h47m44.099536113s) [upload=13164]: Uploading segments... Aug 7 22:27:09.584547339 (12h47m44.099649917s) [upload=13164]: Got next segment. Inline: true Aug 7 22:27:09.584576624 (12h47m44.09967921s) [upload=13164]: Waiting for error group managing segments... Aug 7 22:27:09.584764211 (12h47m44.099866798s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:09.584774497 (12h47m44.099877084s) [upload=13164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:09.584779600 (12h47m44.099882182s) [upload=13164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:09.656944239 (12h47m44.172046803s) [upload=13164]: Done uploading segments... Aug 7 22:27:09.656961099 (12h47m44.172063662s) [upload=13164]: Upload finished. err: <nil> Aug 7 22:28:22.451723369 (12h48m56.966825932s) [upload=13182]: Starting upload Aug 7 22:28:22.451781104 (12h48m56.96688368s) [upload=13182]: Done starting upload Aug 7 22:28:22.451842252 (12h48m56.966944827s) [upload=13182]: Uploading segments... Aug 7 22:28:22.451894112 (12h48m56.9669967s) [upload=13182]: Got next segment. Inline: true Aug 7 22:28:22.451901034 (12h48m56.967003616s) [upload=13182]: Waiting for error group managing segments... Aug 7 22:28:22.451944887 (12h48m56.967047449s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:22.451948845 (12h48m56.967051407s) [upload=13182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:22.451951261 (12h48m56.967053837s) [upload=13182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:52.423300054 (12h49m26.938402631s) [upload=13182]: Done uploading segments... Aug 7 22:28:52.423324837 (12h49m26.938427414s) [upload=13182]: Upload finished. err: metaclient: context deadline exceeded Duration: 225.744403ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [.............XXXXXXXXXXXXXXXXXXXXX.XX.X.....................] duration: 259.124249ms FAIL (2025-08-07 22:45:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 259.155883ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..................XXXXXXXXXXX.XXXXXX........................] duration: 851.921489ms FAIL (2025-08-07 22:45:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 851.994483ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [............XXXXXXXXXXXXXXXXX..X.X..........................] duration: 238.167915ms FAIL (2025-08-07 22:45:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 238.193286ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [...................XXXXXXXXXXX.X..X.X.......................] duration: 227.584094ms FAIL (2025-08-07 22:45:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=13065]: Starting upload Aug 7 22:19:39.328870365 (12h40m13.843972956s) [upload=13065]: Done starting upload Aug 7 22:19:39.329172418 (12h40m13.844274986s) [upload=13065]: Uploading segments... Aug 7 22:19:39.329244607 (12h40m13.844347193s) [upload=13065]: Got next segment. Inline: true Aug 7 22:19:39.329250690 (12h40m13.844353268s) [upload=13065]: Waiting for error group managing segments... Aug 7 22:19:39.329302352 (12h40m13.844404927s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:39.329309321 (12h40m13.844411902s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:39.329313964 (12h40m13.844416551s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:39.300343613 (12h41m13.81544618s) [upload=13065]: Done uploading segments... Aug 7 22:20:39.300363356 (12h41m13.815465922s) [upload=13065]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:45.259131358 (12h41m19.774233925s) [upload=13078]: Starting upload Aug 7 22:20:45.259192437 (12h41m19.774295013s) [upload=13078]: Done starting upload Aug 7 22:20:45.259251322 (12h41m19.774353897s) [upload=13078]: Uploading segments... Aug 7 22:20:45.259289829 (12h41m19.774392414s) [upload=13078]: Got next segment. Inline: true Aug 7 22:20:45.259294478 (12h41m19.774397054s) [upload=13078]: Waiting for error group managing segments... Aug 7 22:20:45.259332525 (12h41m19.774435101s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:45.259337069 (12h41m19.774439635s) [upload=13078]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:45.259340263 (12h41m19.77444284s) [upload=13078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:45.229308083 (12h42m19.744410669s) [upload=13078]: Done uploading segments... Aug 7 22:21:45.229380638 (12h42m19.744483223s) [upload=13078]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:59.915154556 (12h42m34.430257139s) [upload=13097]: Starting upload Aug 7 22:21:59.915242531 (12h42m34.430345108s) [upload=13097]: Done starting upload Aug 7 22:21:59.915358515 (12h42m34.430461092s) [upload=13097]: Uploading segments... Aug 7 22:21:59.915398067 (12h42m34.430500653s) [upload=13097]: Got next segment. Inline: true Aug 7 22:21:59.915403151 (12h42m34.430505727s) [upload=13097]: Waiting for error group managing segments... Aug 7 22:21:59.915458296 (12h42m34.430560863s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:59.915463422 (12h42m34.430565998s) [upload=13097]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:59.915466628 (12h42m34.430569269s) [upload=13097]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:59.883702618 (12h43m34.398805207s) [upload=13097]: Done uploading segments... Aug 7 22:22:59.883744759 (12h43m34.398847337s) [upload=13097]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:00.217702003 (12h43m34.732804585s) [upload=13110]: Starting upload Aug 7 22:23:00.217805899 (12h43m34.732908483s) [upload=13110]: Done starting upload Aug 7 22:23:00.217896494 (12h43m34.732999072s) [upload=13110]: Uploading segments... Aug 7 22:23:00.217970831 (12h43m34.733073418s) [upload=13110]: Got next segment. Inline: true Aug 7 22:23:00.217979414 (12h43m34.733081991s) [upload=13110]: Waiting for error group managing segments... Aug 7 22:23:00.218043817 (12h43m34.733146394s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:00.218051159 (12h43m34.733153734s) [upload=13110]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:00.218054412 (12h43m34.733156988s) [upload=13110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:00.464344185 (12h43m34.979446778s) [upload=13110]: Done uploading segments... Aug 7 22:23:00.464376970 (12h43m34.979479551s) [upload=13110]: Upload finished. err: <nil> Aug 7 22:24:07.143696312 (12h44m41.658798899s) [upload=13125]: Starting upload Aug 7 22:24:07.143811006 (12h44m41.658913589s) [upload=13125]: Done starting upload Aug 7 22:24:07.143914011 (12h44m41.659016577s) [upload=13125]: Uploading segments... Aug 7 22:24:07.143973667 (12h44m41.659076252s) [upload=13125]: Got next segment. Inline: true Aug 7 22:24:07.143979447 (12h44m41.659082034s) [upload=13125]: Waiting for error group managing segments... Aug 7 22:24:07.144045718 (12h44m41.659148297s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:07.144051783 (12h44m41.659154361s) [upload=13125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:07.144057848 (12h44m41.659160429s) [upload=13125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.110905816 (12h45m41.626008396s) [upload=13125]: Done uploading segments... Aug 7 22:25:07.110929433 (12h45m41.626032014s) [upload=13125]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:07.223172827 (12h45m41.738275404s) [upload=13138]: Starting upload Aug 7 22:25:07.223254341 (12h45m41.738356918s) [upload=13138]: Done starting upload Aug 7 22:25:07.223368837 (12h45m41.738471414s) [upload=13138]: Uploading segments... Aug 7 22:25:07.223445919 (12h45m41.738548507s) [upload=13138]: Got next segment. Inline: true Aug 7 22:25:07.223452833 (12h45m41.738555412s) [upload=13138]: Waiting for error group managing segments... Aug 7 22:25:07.223511692 (12h45m41.738614278s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:07.223518073 (12h45m41.738620658s) [upload=13138]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:07.223521493 (12h45m41.738624078s) [upload=13138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:07.315623341 (12h45m41.830725924s) [upload=13138]: Done uploading segments... Aug 7 22:25:07.315640785 (12h45m41.830743361s) [upload=13138]: Upload finished. err: <nil> Aug 7 22:26:03.367436251 (12h46m37.882538842s) [upload=13151]: Starting upload Aug 7 22:26:03.367515051 (12h46m37.882617639s) [upload=13151]: Done starting upload Aug 7 22:26:03.367596048 (12h46m37.882698629s) [upload=13151]: Uploading segments... Aug 7 22:26:03.367644909 (12h46m37.8827475s) [upload=13151]: Got next segment. Inline: true Aug 7 22:26:03.367650467 (12h46m37.882753051s) [upload=13151]: Waiting for error group managing segments... Aug 7 22:26:03.367698612 (12h46m37.882801214s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:03.367705011 (12h46m37.882807591s) [upload=13151]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:03.367709618 (12h46m37.882812202s) [upload=13151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:03.460311519 (12h46m37.975414085s) [upload=13151]: Done uploading segments... Aug 7 22:26:03.460350352 (12h46m37.975452923s) [upload=13151]: Upload finished. err: <nil> Aug 7 22:26:51.117138988 (12h47m25.632241558s) [upload=13162]: Starting upload Aug 7 22:26:51.117235608 (12h47m25.632338187s) [upload=13162]: Done starting upload Aug 7 22:26:51.117347075 (12h47m25.632449638s) [upload=13162]: Uploading segments... Aug 7 22:26:51.117436995 (12h47m25.632539563s) [upload=13162]: Got next segment. Inline: true Aug 7 22:26:51.117443130 (12h47m25.632545707s) [upload=13162]: Waiting for error group managing segments... Aug 7 22:26:51.117504554 (12h47m25.632607133s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:51.117509417 (12h47m25.632611996s) [upload=13162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:51.117512811 (12h47m25.63261539s) [upload=13162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.086979010 (12h48m25.602081586s) [upload=13162]: Done uploading segments... Aug 7 22:27:51.086998672 (12h48m25.602101249s) [upload=13162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:51.193960256 (12h48m25.709062823s) [upload=13178]: Starting upload Aug 7 22:27:51.194024831 (12h48m25.709127407s) [upload=13178]: Done starting upload Aug 7 22:27:51.194084091 (12h48m25.709186657s) [upload=13178]: Uploading segments... Aug 7 22:27:51.194125626 (12h48m25.709228211s) [upload=13178]: Got next segment. Inline: true Aug 7 22:27:51.194132041 (12h48m25.709234607s) [upload=13178]: Waiting for error group managing segments... Aug 7 22:27:51.194172625 (12h48m25.709275192s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:51.194177004 (12h48m25.709279581s) [upload=13178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:51.194180164 (12h48m25.709282731s) [upload=13178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:51.254375304 (12h48m25.769477873s) [upload=13178]: Done uploading segments... Aug 7 22:27:51.254404389 (12h48m25.76950696s) [upload=13178]: Upload finished. err: <nil> Aug 7 22:28:44.417417956 (12h49m18.932520542s) [upload=13188]: Starting upload Aug 7 22:28:44.417530389 (12h49m18.932632976s) [upload=13188]: Done starting upload Aug 7 22:28:44.417616643 (12h49m18.93271922s) [upload=13188]: Uploading segments... Aug 7 22:28:44.417757556 (12h49m18.932860142s) [upload=13188]: Got next segment. Inline: true Aug 7 22:28:44.417764563 (12h49m18.932867158s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:28:44.417818209 (12h49m18.932920789s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:44.417825102 (12h49m18.932927678s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:44.417832015 (12h49m18.932934599s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:37.965879533 (12h50m12.480982114s) [upload=13188]: Done uploading segments... Aug 7 22:29:37.965901992 (12h50m12.481004571s) [upload=13188]: Upload finished. err: <nil> Duration: 227.614164ms core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..XX..............................] duration: 49.889462ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:45:33 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 49.916318ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX..XX.........................X.XX.........................] duration: 463.545635ms FAIL (2025-08-07 22:45:32 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad=12957]: Uploading segments... Aug 7 22:12:11.442470431 (12h32m45.957573001s) [upload=12957]: Got next segment. Inline: false Aug 7 22:12:11.442536399 (12h32m45.957638969s) [upload=12957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.442542497 (12h32m45.957645072s) [upload=12957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:41.409489142 (12h33m15.924591718s) [upload=12957]: Next segment err: context deadline exceeded Aug 7 22:12:41.409585580 (12h33m15.924688156s) [upload=12957]: Done uploading segments... Aug 7 22:12:41.409635672 (12h33m15.92473824s) [upload=12957]: Upload finished. err: context deadline exceeded Aug 7 22:12:41.517765572 (12h33m16.032868143s) [upload=12968]: Starting upload Aug 7 22:12:41.517838005 (12h33m16.032940574s) [upload=12968]: Done starting upload Aug 7 22:12:41.517963005 (12h33m16.033065573s) [upload=12968]: Uploading segments... Aug 7 22:12:41.518335606 (12h33m16.033438184s) [upload=12968]: Got next segment. Inline: false Aug 7 22:12:41.518426941 (12h33m16.033529524s) [upload=12968]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:41.518434836 (12h33m16.03353742s) [upload=12968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:48.560277241 (12h33m23.075379809s) [seg_pos={0 0}, upload=12968]: Begin upload segment... Aug 7 22:12:48.560321663 (12h33m23.075424233s) [seg_pos={0 0}, upload=12968]: Done begin upload segment. Aug 7 22:12:48.560346254 (12h33m23.075448833s) [noise=true , address=77.90.31.21:4878 , piece_id=AYEE7NAJIYWU72ZKPBR44ZRU7T7HUB4FPAHH6V3KYKBSULV3AJMQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.560385235 (12h33m23.075487817s) [noise=true , address=104.36.82.4:4802 , piece_id=WUKRWI3KJKXS2KBP5WRMPI2UGYT7RKQ6KBBOAPPW6Z5ISN72VIWQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.560895927 (12h33m23.075998514s) [noise=true , address=45.91.116.5:4297 , piece_id=5TO5AKA4MGTPNPH6VWPA4GIDECUKKPFWSR5VZ3263TYANZ4BZGPA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.561270415 (12h33m23.076372997s) [noise=true , address=45.91.116.9:4262 , piece_id=UGR7YOFEX6MXTFNZPJ7EKO5DH3V7R47FOKI5ZNAHB4ZYH7J7UHKA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.562228406 (12h33m23.077330973s) [noise=true , address=45.91.116.11:4117 , piece_id=PQYC55OVQBT5EFFWHBMXXDFBGCDNBZFAYJR42IK5JEDDAXXWDJYA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.563480004 (12h33m23.078582611s) [noise=true , address=45.91.116.14:4190 , piece_id=3U7NYGD2JPOBZAHNJ2GUNUA3GE6JYRLJBMYQTEREUVWBR6QWAM6A, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.563854062 (12h33m23.078956648s) [noise=true , address=45.91.116.11:4118 , piece_id=A5SGTLAHM6E3N7AHB7U3KGZFYHDKO4JBE2IHNH76YKJUR54TGMZA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.564192678 (12h33m23.079295261s) [noise=true , address=104.36.82.12:4151 , piece_id=BV2GWAIQGUQ3BKVWHDBQ2RK772VQTLGYJ4WZU5VPALAGO7VOAZMA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.564542455 (12h33m23.079645037s) [noise=true , address=77.90.31.17:4771 , piece_id=OZMRVKETNTDWOLXUFP3IJLGQENMWSWB3MOTJKXXRUEDQHOFV6LQA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565038234 (12h33m23.080140824s) [noise=true , address=77.90.31.16:4733 , piece_id=LB4CQZC5X7QUEYL5OAQ4A4MA6OFO6LPC7RKCLN5AT6V6UGFHAKEQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565530690 (12h33m23.08063327s) [noise=true , address=77.90.31.10:4520 , piece_id=BKGMZCYXKDUXT42DSCXXLBKMYASXQWCG43HN4NJT76A5EESUNI3Q, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.565683172 (12h33m23.080785758s) [noise=true , address=45.91.116.6:4975 , piece_id=KALV4QK4ML56PQUNJFDQXHTXZ5VOJYWZMXQNGHA6U5FLBXEW46TA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566145847 (12h33m23.081248429s) [noise=true , address=77.90.31.11:4553 , piece_id=TY7MWYZNDLY7CMDYALZFEYN2HE64TUEYW522DJKBFL463MMQIPOA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566452821 (12h33m23.0815554s) [noise=true , address=77.90.31.18:4805 , piece_id=NP4IZUJHXPID26UBGUMTHYRM2TEDF2BHPAE5Z4B34OVHJ4RRX5LA, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.566765755 (12h33m23.081868318s) [noise=true , address=77.90.31.21:4877 , piece_id=53XKORZS4TOBNGN2VM7DF55MS4NCZH7NX7F4ZZHDMJCKNT3YPVZQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.567074238 (12h33m23.082176801s) [noise=true , address=104.36.82.6:4876 , piece_id=MFX6SCYKRA4PAZ4VJ3JMKK6THX5MOCE4SBENR6NXDLOWA7C2GGJQ, seg_pos={0 0}, upload=12968]: Uploading piece... Aug 7 22:12:48.674430176 (12h33m23.189532754s) [upload=12968]: Next returned nil segment Aug 7 22:12:48.674443729 (12h33m23.189546305s) [upload=12968]: Waiting for error group managing segments... Aug 7 22:12:48.677024873 (12h33m23.192127438s) [noise=true , address=77.90.31.11:4553 , piece_id=TY7MWYZNDLY7CMDYALZFEYN2HE64TUEYW522DJKBFL463MMQIPOA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.677316512 (12h33m23.192419097s) [noise=true , address=45.91.116.5:4297 , piece_id=5TO5AKA4MGTPNPH6VWPA4GIDECUKKPFWSR5VZ3263TYANZ4BZGPA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.677774329 (12h33m23.192876899s) [noise=true , address=45.91.116.9:4262 , piece_id=UGR7YOFEX6MXTFNZPJ7EKO5DH3V7R47FOKI5ZNAHB4ZYH7J7UHKA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678157641 (12h33m23.193260219s) [noise=true , address=45.91.116.11:4118 , piece_id=A5SGTLAHM6E3N7AHB7U3KGZFYHDKO4JBE2IHNH76YKJUR54TGMZA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678319308 (12h33m23.193421874s) [noise=true , address=45.91.116.11:4117 , piece_id=PQYC55OVQBT5EFFWHBMXXDFBGCDNBZFAYJR42IK5JEDDAXXWDJYA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678828950 (12h33m23.193931524s) [noise=true , address=45.91.116.6:4975 , piece_id=KALV4QK4ML56PQUNJFDQXHTXZ5VOJYWZMXQNGHA6U5FLBXEW46TA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.678921127 (12h33m23.194023707s) [noise=true , address=45.91.116.14:4190 , piece_id=3U7NYGD2JPOBZAHNJ2GUNUA3GE6JYRLJBMYQTEREUVWBR6QWAM6A, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.679383015 (12h33m23.194485578s) [noise=true , address=77.90.31.21:4877 , piece_id=53XKORZS4TOBNGN2VM7DF55MS4NCZH7NX7F4ZZHDMJCKNT3YPVZQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.679808178 (12h33m23.194910761s) [noise=true , address=77.90.31.18:4805 , piece_id=NP4IZUJHXPID26UBGUMTHYRM2TEDF2BHPAE5Z4B34OVHJ4RRX5LA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.680957829 (12h33m23.196060398s) [noise=true , address=77.90.31.21:4878 , piece_id=AYEE7NAJIYWU72ZKPBR44ZRU7T7HUB4FPAHH6V3KYKBSULV3AJMQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.681390098 (12h33m23.196492675s) [noise=true , address=104.36.82.4:4802 , piece_id=WUKRWI3KJKXS2KBP5WRMPI2UGYT7RKQ6KBBOAPPW6Z5ISN72VIWQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.681868182 (12h33m23.19697076s) [noise=true , address=77.90.31.16:4733 , piece_id=LB4CQZC5X7QUEYL5OAQ4A4MA6OFO6LPC7RKCLN5AT6V6UGFHAKEQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.682434566 (12h33m23.197537204s) [noise=true , address=77.90.31.17:4771 , piece_id=OZMRVKETNTDWOLXUFP3IJLGQENMWSWB3MOTJKXXRUEDQHOFV6LQA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: <nil> Aug 7 22:12:48.682445484 (12h33m23.197548047s) [seg_pos={0 0}, upload=12968]: Segment reached optimal threshold of 13 pieces. Aug 7 22:12:48.682659606 (12h33m23.197762177s) [noise=true , address=77.90.31.10:4520 , piece_id=BKGMZCYXKDUXT42DSCXXLBKMYASXQWCG43HN4NJT76A5EESUNI3Q, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Aug 7 22:12:48.682743670 (12h33m23.197846239s) [noise=true , address=104.36.82.12:4151 , piece_id=BV2GWAIQGUQ3BKVWHDBQ2RK772VQTLGYJ4WZU5VPALAGO7VOAZMA, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection; write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43886->104.36.82.12:4151: use of closed network connection Aug 7 22:12:48.682881386 (12h33m23.197983962s) [noise=true , address=104.36.82.6:4876 , piece_id=MFX6SCYKRA4PAZ4VJ3JMKK6THX5MOCE4SBENR6NXDLOWA7C2GGJQ, seg_pos={0 0}, upload=12968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: read tcp 159.223.143.39:35642->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 22:12:48.682925198 (12h33m23.198027778s) [seg_pos={0 0}, upload=12968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:12:48.683071518 (12h33m23.198174098s) [upload=12968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:12:48.683078012 (12h33m23.19818059s) [upload=12968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:26.914480867 (12h50m1.429583435s) [upload=12968]: Done uploading segments... Aug 7 22:29:26.914503234 (12h50m1.42960581s) [upload=12968]: Upload finished. err: metaclient: context deadline exceeded Duration: 463.579788ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................XXXXXXXXX.X.......................] duration: 1.784970929s FAIL (2025-08-07 22:45:29 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.785014237s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)