Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 319646 reports (Page 299 of 6393)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..........X] duration: 1m3.218818421s trace: https://jaeger.dev.storj.io/trace/7b00c8e570527591 FAIL (2026-03-18 19:45:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:44:44.531335519 (296h44m42.128106551s) [upload=332380]: Starting upload Mar 18 19:44:44.531475325 (296h44m42.128246356s) [upload=332380]: Done starting upload Mar 18 19:44:44.531632499 (296h44m42.128403513s) [upload=332380]: Uploading segments... Mar 18 19:44:44.531858554 (296h44m42.128629577s) [upload=332380]: Got next segment. Inline: false Mar 18 19:44:44.531933224 (296h44m42.128704248s) [upload=332380]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:44:44.531939737 (296h44m42.12871076s) [upload=332380]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:44:48.315984880 (296h44m45.912755906s) [seg_pos={0 0}, upload=332380]: Begin upload segment... Mar 18 19:44:48.316064996 (296h44m45.91283602s) [seg_pos={0 0}, upload=332380]: Done begin upload segment. Mar 18 19:44:48.316091356 (296h44m45.912862388s) [noise=true , address=193.149.164.69:4086 , piece_id=64I67SQ7XGQZTXXIM3QEUOZJ3MR4PNYTX2566KYO6XWTKV42N5EA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.316109641 (296h44m45.91288067s) [noise=true , address=167.17.72.69:4728 , piece_id=FIJM4A53FLXYPIP2FC6BMAGKXNLAOCUANADBPNSTWBH6MUXUHTAQ, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.316713887 (296h44m45.913484914s) [noise=true , address=193.149.164.67:4006 , piece_id=3UBN2ZFDDTC4MEMMCYXZIUW6BGB4XVHS2QRP4UIZULWO35NKSCHQ, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.317160473 (296h44m45.913931499s) [noise=true , address=79.127.219.38:4827 , piece_id=YRVFKYNAINZHHTMFXFS7TGH3M34HI4YOYK3GUOJRCWC6MAG7KMNA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.317720426 (296h44m45.914491456s) [noise=true , address=145.79.227.36:4541 , piece_id=JCQKPMUEG5DXBBMQCQUOW2DI2BZSOEBPXMIR7EGUBFATY33FYJTA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.318186378 (296h44m45.914957405s) [noise=true , address=79.127.219.46:4911 , piece_id=F2GG2E6Q7FJDC5W5ZHAWGPCPQ3BKC5KS4UXCXW2ZHGITTZ7K5WJQ, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.318833541 (296h44m45.915604573s) [noise=true , address=45.91.116.13:4179 , piece_id=3QXFPDPFUF6UGPXA62QOS3EQZAYCN4R62XYPU45JCBYN35ZKXNXA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.319762098 (296h44m45.916533136s) [noise=true , address=77.90.31.14:4671 , piece_id=5PBLF7RDWNO2XKERHHVQYQNSPXGS7UNDB4P3U7YZGXKZGS54LK7A, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.321366156 (296h44m45.918137183s) [noise=true , address=145.79.227.51:4423 , piece_id=NCU3O5HXGX5TTRKKKHGUMTOISBNMQQDBJJTYAAPEF7XFBBDMYPHQ, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.323285059 (296h44m45.920056091s) [noise=true , address=145.79.227.52:4474 , piece_id=LGOA3K4B53T7G7ZK55WWEINYAOH3C2PKPRNMYNJWRDRE7UXFQY7A, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.325173105 (296h44m45.921944138s) [noise=true , address=145.79.227.41:4100 , piece_id=OXEH6J5B5Y7TFVEBJNMTXN7FJIVU26SQP7UTHY727HBOHOIPASVA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.326747430 (296h44m45.923518465s) [noise=true , address=79.127.205.235:4263 , piece_id=NOFZRFE2HLK3IQS53AMY3QMBLVQLG5E5QAWVW65JOWSQK7KS6GUQ, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.327575272 (296h44m45.924346296s) [noise=true , address=77.90.31.10:4538 , piece_id=2ZQYIG3VRJRY3TKHGC4MMGEHGSOJ6OTFSUVFZQJEUXLK4P6UHXIA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.329708619 (296h44m45.926479652s) [noise=true , address=77.90.31.18:4839 , piece_id=FPRXT44J3P4H5Z6U25XOA2IRAKE37SWZJZGYT2NSSRQU2HPUXPWA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.330683776 (296h44m45.927454809s) [noise=true , address=193.149.164.68:4041 , piece_id=NU2QPZUVQGH4QIRS4Y2VVJMTTHKCAYHU7JPGBXZJ7TQ6D3C2MRGA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.333760099 (296h44m45.930531133s) [noise=true , address=45.91.116.4:4960 , piece_id=HLGV3OIX6AOSTDGXPZI3CLZQFPT76JRIN2MOHZBBGE4IUNBLNACA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.349633456 (296h44m45.946404488s) [noise=true , address=45.91.116.8:4050 , piece_id=66XX4N57QYWBEUTGYMJCMAFV5H4Q4KFCGBKJR2VJ4DFQF5ZBLMTQ, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.350929519 (296h44m45.947700547s) [noise=true , address=145.79.227.44:4214 , piece_id=NMUNQCAFF36ERUWCRJW7HQLRZL32VKVE3VVKCYUECXC45YBEJQDA, seg_pos={0 0}, upload=332380]: Uploading piece... Mar 18 19:44:48.394353954 (296h44m45.991124984s) [upload=332380]: Next returned nil segment Mar 18 19:44:48.394371623 (296h44m45.99114266s) [upload=332380]: Waiting for error group managing segments... Mar 18 19:44:48.403480517 (296h44m46.000251541s) [noise=true , address=79.127.219.46:4911 , piece_id=F2GG2E6Q7FJDC5W5ZHAWGPCPQ3BKC5KS4UXCXW2ZHGITTZ7K5WJQ, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.410246355 (296h44m46.007017379s) [noise=true , address=77.90.31.14:4671 , piece_id=5PBLF7RDWNO2XKERHHVQYQNSPXGS7UNDB4P3U7YZGXKZGS54LK7A, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.412238585 (296h44m46.009009614s) [noise=true , address=45.91.116.4:4960 , piece_id=HLGV3OIX6AOSTDGXPZI3CLZQFPT76JRIN2MOHZBBGE4IUNBLNACA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.413090726 (296h44m46.00986175s) [noise=true , address=77.90.31.10:4538 , piece_id=2ZQYIG3VRJRY3TKHGC4MMGEHGSOJ6OTFSUVFZQJEUXLK4P6UHXIA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.413676024 (296h44m46.010447049s) [noise=true , address=77.90.31.18:4839 , piece_id=FPRXT44J3P4H5Z6U25XOA2IRAKE37SWZJZGYT2NSSRQU2HPUXPWA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.417145192 (296h44m46.013916215s) [noise=true , address=45.91.116.13:4179 , piece_id=3QXFPDPFUF6UGPXA62QOS3EQZAYCN4R62XYPU45JCBYN35ZKXNXA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.438259257 (296h44m46.035030282s) [noise=true , address=79.127.219.38:4827 , piece_id=YRVFKYNAINZHHTMFXFS7TGH3M34HI4YOYK3GUOJRCWC6MAG7KMNA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.516935895 (296h44m46.113706918s) [noise=true , address=45.91.116.8:4050 , piece_id=66XX4N57QYWBEUTGYMJCMAFV5H4Q4KFCGBKJR2VJ4DFQF5ZBLMTQ, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.750462656 (296h44m46.347233684s) [noise=true , address=167.17.72.69:4728 , piece_id=FIJM4A53FLXYPIP2FC6BMAGKXNLAOCUANADBPNSTWBH6MUXUHTAQ, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.926883730 (296h44m46.523654754s) [noise=true , address=193.149.164.69:4086 , piece_id=64I67SQ7XGQZTXXIM3QEUOZJ3MR4PNYTX2566KYO6XWTKV42N5EA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.932608316 (296h44m46.52937934s) [noise=true , address=79.127.205.235:4263 , piece_id=NOFZRFE2HLK3IQS53AMY3QMBLVQLG5E5QAWVW65JOWSQK7KS6GUQ, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.948627380 (296h44m46.545398406s) [noise=true , address=193.149.164.67:4006 , piece_id=3UBN2ZFDDTC4MEMMCYXZIUW6BGB4XVHS2QRP4UIZULWO35NKSCHQ, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:44:48.960357900 (296h44m46.557128923s) [noise=true , address=193.149.164.68:4041 , piece_id=NU2QPZUVQGH4QIRS4Y2VVJMTTHKCAYHU7JPGBXZJ7TQ6D3C2MRGA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: <nil> Mar 18 19:45:44.413232108 (296h45m42.01000314s) [noise=true , address=145.79.227.51:4423 , piece_id=NCU3O5HXGX5TTRKKKHGUMTOISBNMQQDBJJTYAAPEF7XFBBDMYPHQ, seg_pos={0 0}, upload=332380]: Done uploading piece. err: ecclient: upload failed (node:12fUb7FPBapbEwdkCE7vsgwcbF5B8KGcDVZAtihDB9SDuvUpzA5, address:145.79.227.51:4423): protocol: rpc: dial tcp 145.79.227.51:4423: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:44.413278812 (296h45m42.010049841s) [noise=true , address=145.79.227.41:4100 , piece_id=OXEH6J5B5Y7TFVEBJNMTXN7FJIVU26SQP7UTHY727HBOHOIPASVA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:145.79.227.41:4100): protocol: rpc: dial tcp 145.79.227.41:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:44.413562353 (296h45m42.010333388s) [noise=true , address=145.79.227.44:4214 , piece_id=NMUNQCAFF36ERUWCRJW7HQLRZL32VKVE3VVKCYUECXC45YBEJQDA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: ecclient: upload failed (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi, address:145.79.227.44:4214): protocol: rpc: dial tcp 145.79.227.44:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:44.413661393 (296h45m42.010432422s) [noise=true , address=145.79.227.36:4541 , piece_id=JCQKPMUEG5DXBBMQCQUOW2DI2BZSOEBPXMIR7EGUBFATY33FYJTA, seg_pos={0 0}, upload=332380]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:145.79.227.36:4541): protocol: rpc: dial tcp 145.79.227.36:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:44.413723462 (296h45m42.010494495s) [noise=true , address=145.79.227.52:4474 , piece_id=LGOA3K4B53T7G7ZK55WWEINYAOH3C2PKPRNMYNJWRDRE7UXFQY7A, seg_pos={0 0}, upload=332380]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:145.79.227.52:4474): protocol: rpc: dial tcp 145.79.227.52:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:44.413757860 (296h45m42.010528884s) [seg_pos={0 0}, upload=332380]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:45:47.629192408 (296h45m45.225963438s) [upload=332380]: Done uploading segments... Mar 18 19:45:47.629238634 (296h45m45.22600966s) [upload=332380]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m3.218845224s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 30.04032909s trace: https://jaeger.dev.storj.io/trace/653a0c35ab2af8cd FAIL (2026-03-18 19:45:36 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Mar 18 19:45:06.524064031 (296h45m4.120835096s) [upload=332388]: Starting upload Mar 18 19:45:06.524168668 (296h45m4.120939704s) [upload=332388]: Done starting upload Mar 18 19:45:06.524531602 (296h45m4.121302616s) [upload=332388]: Uploading segments... Mar 18 19:45:06.524601222 (296h45m4.121372258s) [upload=332388]: Got next segment. Inline: false Mar 18 19:45:06.524653588 (296h45m4.121424623s) [upload=332388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:45:06.524660524 (296h45m4.121431557s) [upload=332388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:45:06.992463883 (296h45m4.589234916s) [seg_pos={0 0}, upload=332388]: Begin upload segment... Mar 18 19:45:06.992535067 (296h45m4.589306096s) [seg_pos={0 0}, upload=332388]: Done begin upload segment. Mar 18 19:45:06.992703297 (296h45m4.589474329s) [noise=true , address=45.91.116.8:4049 , piece_id=QSTYZCVJQBGCYKNGUITCZGMES7IIYYCOOLHGJOEU5UECQNV5BQ2A, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.992819441 (296h45m4.589590464s) [noise=true , address=145.79.227.28:4731 , piece_id=4FW7EISFZBDHHLEN3P3IUO6WNW3KKAJUTSD2LRMJFXKUUVKMW5CQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.993492595 (296h45m4.590263627s) [noise=true , address=145.79.227.21:4479 , piece_id=AGTVPSLGEPDSHAOWWQ4XXKYEVUXLQKG77IOTLIEIFNXXW473FCTA, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.994165913 (296h45m4.590936946s) [noise=true , address=45.91.116.4:4952 , piece_id=ETKLETI4DNWKDYRJV7P25KRL66VMNK7V2NHUGNV7PVB6WM3UFWXQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.994255406 (296h45m4.591026436s) [noise=true , address=45.91.116.10:4083 , piece_id=GJNYBCR6DPC4YTANJOQQWW6DL7V57KY6B25D4HARSWOXXILKMYKQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.995047698 (296h45m4.591818732s) [noise=true , address=77.90.31.12:4594 , piece_id=X4HR32BMITIHKUNQBJPDKRS6GKV7O53LSMTNOHZLGLIXE4H4JSCA, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.995241316 (296h45m4.592012348s) [noise=true , address=77.90.31.21:4878 , piece_id=MUHMH4RR4ED6YB4L6GKG5FJEVD7DLS2ZZLNHR6QZMJ7R74KF45BA, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.995688459 (296h45m4.592459494s) [noise=true , address=77.90.31.17:4774 , piece_id=HMZQ5OTLJBBGW2B7X6CEFF5FPRUECDYOF2QF7BPPZ5JQASK4RSJA, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.996130529 (296h45m4.592901553s) [noise=true , address=45.91.116.6:4976 , piece_id=272ZAWYCAS6IDQ5OSE7NELDPLDAH4XWYWADC6DPTTPLGENQTNE6Q, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.996279427 (296h45m4.593050475s) [noise=true , address=145.79.227.5:4225 , piece_id=77L7C2AMM4DYZV24TSPPIKMNKRH7UIX6CISJU6ANMLIIQEVOQGHQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.996690253 (296h45m4.593461283s) [noise=true , address=145.79.227.27:4694 , piece_id=NLLW53KMGI5FPNELZJZNZ5RUMHM3DMYKFRIPJMDXF4MNFANWY6TQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.997274414 (296h45m4.594045457s) [noise=true , address=45.91.116.9:4264 , piece_id=ZPSISFUXHKVKL3B25ZKIGLY46DQ5RLURQZLHOBWBOQ25GUWNWMEQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.997949546 (296h45m4.594720581s) [noise=true , address=79.127.219.41:4929 , piece_id=O2YKGED5QJGPJRGK2NM3ZKKGPDBCVE3HJ6UDH2J3YYNOTVTLDGRQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.998567617 (296h45m4.595338643s) [noise=true , address=45.91.116.11:4118 , piece_id=DZU4UGSV3OYA6U6O5HX6Y3WDSPSZCBOJO3T4PVZDD7QNYB7VZIWQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:06.999219513 (296h45m4.595990544s) [noise=true , address=145.79.227.50:4373 , piece_id=E52KRH37FOBHRFDNON6OLXNX3MKSZWA6TZV7D5GPMLQXSBJBRFYQ, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:07.000163955 (296h45m4.596934982s) [noise=true , address=79.127.219.38:4821 , piece_id=HOWEPV3JGBYV7FY7HCVWK5HKIUO36SNUEQ32HET44ADDO337AJMA, seg_pos={0 0}, upload=332388]: Uploading piece... Mar 18 19:45:07.084981771 (296h45m4.681752794s) [upload=332388]: Next returned nil segment Mar 18 19:45:07.084995705 (296h45m4.681766728s) [upload=332388]: Waiting for error group managing segments... Mar 18 19:45:07.089832934 (296h45m4.68660396s) [noise=true , address=45.91.116.8:4049 , piece_id=QSTYZCVJQBGCYKNGUITCZGMES7IIYYCOOLHGJOEU5UECQNV5BQ2A, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.090358098 (296h45m4.687129137s) [noise=true , address=45.91.116.9:4264 , piece_id=ZPSISFUXHKVKL3B25ZKIGLY46DQ5RLURQZLHOBWBOQ25GUWNWMEQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.090505858 (296h45m4.687276892s) [noise=true , address=45.91.116.4:4952 , piece_id=ETKLETI4DNWKDYRJV7P25KRL66VMNK7V2NHUGNV7PVB6WM3UFWXQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.091272825 (296h45m4.68804385s) [noise=true , address=45.91.116.6:4976 , piece_id=272ZAWYCAS6IDQ5OSE7NELDPLDAH4XWYWADC6DPTTPLGENQTNE6Q, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.091608083 (296h45m4.688379109s) [noise=true , address=79.127.219.41:4929 , piece_id=O2YKGED5QJGPJRGK2NM3ZKKGPDBCVE3HJ6UDH2J3YYNOTVTLDGRQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.093054437 (296h45m4.68982546s) [noise=true , address=77.90.31.21:4878 , piece_id=MUHMH4RR4ED6YB4L6GKG5FJEVD7DLS2ZZLNHR6QZMJ7R74KF45BA, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.095559678 (296h45m4.692330711s) [noise=true , address=79.127.219.38:4821 , piece_id=HOWEPV3JGBYV7FY7HCVWK5HKIUO36SNUEQ32HET44ADDO337AJMA, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.102930048 (296h45m4.699701081s) [noise=true , address=45.91.116.11:4118 , piece_id=DZU4UGSV3OYA6U6O5HX6Y3WDSPSZCBOJO3T4PVZDD7QNYB7VZIWQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.110081001 (296h45m4.706852025s) [noise=true , address=77.90.31.17:4774 , piece_id=HMZQ5OTLJBBGW2B7X6CEFF5FPRUECDYOF2QF7BPPZ5JQASK4RSJA, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.110250213 (296h45m4.707021237s) [noise=true , address=45.91.116.10:4083 , piece_id=GJNYBCR6DPC4YTANJOQQWW6DL7V57KY6B25D4HARSWOXXILKMYKQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:07.223936786 (296h45m4.82070783s) [noise=true , address=77.90.31.12:4594 , piece_id=X4HR32BMITIHKUNQBJPDKRS6GKV7O53LSMTNOHZLGLIXE4H4JSCA, seg_pos={0 0}, upload=332388]: Done uploading piece. err: <nil> Mar 18 19:45:36.489195226 (296h45m34.08596626s) [noise=true , address=145.79.227.28:4731 , piece_id=4FW7EISFZBDHHLEN3P3IUO6WNW3KKAJUTSD2LRMJFXKUUVKMW5CQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: ecclient: upload failed (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK, address:145.79.227.28:4731): protocol: rpc: dial tcp 145.79.227.28:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:36.489222606 (296h45m34.08599364s) [noise=true , address=145.79.227.50:4373 , piece_id=E52KRH37FOBHRFDNON6OLXNX3MKSZWA6TZV7D5GPMLQXSBJBRFYQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: rpc: dial tcp 145.79.227.50:4373: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:36.489557236 (296h45m34.086328271s) [noise=true , address=145.79.227.27:4694 , piece_id=NLLW53KMGI5FPNELZJZNZ5RUMHM3DMYKFRIPJMDXF4MNFANWY6TQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: ecclient: upload failed (node:17sHakGWopKRax7hbzezZASaWtH5tUUdFMP5diZDK3UwP153cd, address:145.79.227.27:4694): protocol: rpc: dial tcp 145.79.227.27:4694: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:36.489695487 (296h45m34.086466517s) [noise=true , address=145.79.227.21:4479 , piece_id=AGTVPSLGEPDSHAOWWQ4XXKYEVUXLQKG77IOTLIEIFNXXW473FCTA, seg_pos={0 0}, upload=332388]: Done uploading piece. err: ecclient: upload failed (node:12VCopFEoHk99GsQNQPuWvTQZRD1bpXi8HHLKXaXRdvFThTxmAk, address:145.79.227.21:4479): protocol: rpc: dial tcp 145.79.227.21:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:36.489996517 (296h45m34.086767582s) [noise=true , address=145.79.227.5:4225 , piece_id=77L7C2AMM4DYZV24TSPPIKMNKRH7UIX6CISJU6ANMLIIQEVOQGHQ, seg_pos={0 0}, upload=332388]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: rpc: dial tcp 145.79.227.5:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:36.490036267 (296h45m34.086807295s) [seg_pos={0 0}, upload=332388]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Mar 18 19:45:36.527108681 (296h45m34.123879704s) [upload=332388]: Done uploading segments... Mar 18 19:45:36.527159563 (296h45m34.123930586s) [upload=332388]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.040347296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XXXXXX.XX.....] duration: 1.086545597s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:45:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.086578149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.......................................XXXXXXXXX.XXX...XXXXX] duration: 5.000708954s trace: https://jaeger.dev.storj.io/trace/7a85afb8b7d2821 FAIL (2026-03-18 19:45:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000789718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [........................................XXXXXXXXX.XXX...XXXX] duration: 5.001126388s trace: https://jaeger.dev.storj.io/trace/58945d915c37ae77 FAIL (2026-03-18 19:45:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001157748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.055252568s trace: https://jaeger.dev.storj.io/trace/5daad7cd887f4854 FAIL (2026-03-18 19:45:06 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 19:44:36.405292961 (296h44m34.002063995s) [upload=332377]: Starting upload Mar 18 19:44:36.405395413 (296h44m34.002166446s) [upload=332377]: Done starting upload Mar 18 19:44:36.405499247 (296h44m34.002270269s) [upload=332377]: Uploading segments... Mar 18 19:44:36.405761145 (296h44m34.002532168s) [upload=332377]: Got next segment. Inline: false Mar 18 19:44:36.405823923 (296h44m34.002594947s) [upload=332377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:44:36.405849261 (296h44m34.002620284s) [upload=332377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:44:36.438112703 (296h44m34.034883727s) [seg_pos={0 0}, upload=332377]: Begin upload segment... Mar 18 19:44:36.438183795 (296h44m34.03495482s) [seg_pos={0 0}, upload=332377]: Done begin upload segment. Mar 18 19:44:36.438251927 (296h44m34.035022951s) [noise=true , address=45.91.116.11:4121 , piece_id=V7EIYE5MOJTAIWNPJNJCQ7GJZGUSIBA6J5OHW32BWNGCBB6UPIFQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.439220535 (296h44m34.035991623s) [noise=true , address=79.127.219.46:4892 , piece_id=43OGXQFPN5XLXCLIBTLMDTK3MS4QMODYMHJLF7MRE4P6OBWAQ3PQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.439288044 (296h44m34.036059075s) [noise=true , address=145.79.227.11:4735 , piece_id=P5CEXRIZW5HYKG6PM2FZ2637DPCFL57R3AC3NNYGSLENYEGVUX6A, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.439784379 (296h44m34.036555404s) [noise=true , address=77.90.31.15:4700 , piece_id=KAZQM3JOKDGB5KYMRQBN2FA446SILIT3PWMA5IGFU3BCQLFZ3T6A, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.439888317 (296h44m34.036659351s) [noise=true , address=145.79.227.41:4087 , piece_id=NS4H2CPABLOBYU54TNIDFNJLKKKVCHR43EYBBBN3KWUAFL34BL5A, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.440198413 (296h44m34.036969438s) [noise=true , address=145.79.227.2:4115 , piece_id=P3OEN6MLRVOH5JTV52QPQ2QQNTLVAK46XU2RTPQ2TAWZ3SEFFQEQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.440325765 (296h44m34.037096791s) [noise=true , address=45.91.116.10:4082 , piece_id=YL3B6LUNPAWC5HH6JJRURBJ5FNGDIPR2LK2FRYE4ZPE5OFBAXCSA, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.440615121 (296h44m34.037386147s) [noise=true , address=145.79.227.51:4414 , piece_id=7ZSATDNULGY5IM6V4IU7H6CC7FX3Q7NZFTSHNXINCR2ZL2JRZWXA, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.440743426 (296h44m34.037514442s) [noise=true , address=145.79.227.5:4224 , piece_id=DXXZ7MSO75W5GUBBERBFQCERX6JO6ODUCOK7KKUNFX2EWI5WZMYQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.440996029 (296h44m34.037767058s) [noise=true , address=77.90.31.12:4590 , piece_id=X45547J4AMBJC32Z55FCY64ICCGVW5V73LAW3XQJ4ECVSSHZDJWQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.441219063 (296h44m34.037990091s) [noise=true , address=45.91.116.13:4157 , piece_id=VJMWBA7KDPVHZ7TDFJ7AXXAX7BSPTGIXPGTE65IKMRKFYPGQN7KA, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.441606967 (296h44m34.038377996s) [noise=true , address=77.90.31.10:4517 , piece_id=C4W7A63YJ7HENC6LRQLPETT7CDDKFIZF5TTJCYFWTWPD674IYVWA, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.442105398 (296h44m34.038876412s) [noise=true , address=145.79.227.14:4839 , piece_id=4LFKQFFLBS2B4GANRSGF2WDFERCYEWVL3U3SKUHTNR4U3CSFF3BQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.442570482 (296h44m34.039341517s) [noise=true , address=45.91.116.12:4255 , piece_id=ZAIWKA5CES2T6FQZDW6FIZ6ZMSFOUMMLH545FB3MHKSMAKQ7U65Q, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.443007414 (296h44m34.039778445s) [noise=true , address=45.91.116.9:4295 , piece_id=UC34P7TGJPXAPV6LT4ZR575GOJ2BMQMTAENYYQXGPU7HJ3N3BAAQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.443508499 (296h44m34.04027953s) [noise=true , address=145.79.227.26:4658 , piece_id=SPDHZLEOEWFL6JYHGQLO23F4WUTZXLM7P75XMPI4MG3UF2PWEPOQ, seg_pos={0 0}, upload=332377]: Uploading piece... Mar 18 19:44:36.510846547 (296h44m34.10761756s) [upload=332377]: Next returned nil segment Mar 18 19:44:36.510861539 (296h44m34.107632562s) [upload=332377]: Waiting for error group managing segments... Mar 18 19:44:36.514677652 (296h44m34.111448666s) [noise=true , address=45.91.116.10:4082 , piece_id=YL3B6LUNPAWC5HH6JJRURBJ5FNGDIPR2LK2FRYE4ZPE5OFBAXCSA, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.515723807 (296h44m34.112494836s) [noise=true , address=45.91.116.11:4121 , piece_id=V7EIYE5MOJTAIWNPJNJCQ7GJZGUSIBA6J5OHW32BWNGCBB6UPIFQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.515954612 (296h44m34.112725635s) [noise=true , address=45.91.116.9:4295 , piece_id=UC34P7TGJPXAPV6LT4ZR575GOJ2BMQMTAENYYQXGPU7HJ3N3BAAQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.516621437 (296h44m34.113392451s) [noise=true , address=45.91.116.13:4157 , piece_id=VJMWBA7KDPVHZ7TDFJ7AXXAX7BSPTGIXPGTE65IKMRKFYPGQN7KA, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.520572109 (296h44m34.117343137s) [noise=true , address=79.127.219.46:4892 , piece_id=43OGXQFPN5XLXCLIBTLMDTK3MS4QMODYMHJLF7MRE4P6OBWAQ3PQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.526746645 (296h44m34.123517669s) [noise=true , address=77.90.31.12:4590 , piece_id=X45547J4AMBJC32Z55FCY64ICCGVW5V73LAW3XQJ4ECVSSHZDJWQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.531588690 (296h44m34.128359718s) [noise=true , address=45.91.116.12:4255 , piece_id=ZAIWKA5CES2T6FQZDW6FIZ6ZMSFOUMMLH545FB3MHKSMAKQ7U65Q, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.532831450 (296h44m34.129602475s) [noise=true , address=77.90.31.10:4517 , piece_id=C4W7A63YJ7HENC6LRQLPETT7CDDKFIZF5TTJCYFWTWPD674IYVWA, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:44:36.542295158 (296h44m34.139066181s) [noise=true , address=77.90.31.15:4700 , piece_id=KAZQM3JOKDGB5KYMRQBN2FA446SILIT3PWMA5IGFU3BCQLFZ3T6A, seg_pos={0 0}, upload=332377]: Done uploading piece. err: <nil> Mar 18 19:45:06.395659920 (296h45m3.992430955s) [noise=true , address=145.79.227.26:4658 , piece_id=SPDHZLEOEWFL6JYHGQLO23F4WUTZXLM7P75XMPI4MG3UF2PWEPOQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr, address:145.79.227.26:4658): protocol: rpc: dial tcp 145.79.227.26:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.396201463 (296h45m3.992972496s) [noise=true , address=145.79.227.11:4735 , piece_id=P5CEXRIZW5HYKG6PM2FZ2637DPCFL57R3AC3NNYGSLENYEGVUX6A, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS, address:145.79.227.11:4735): protocol: rpc: dial tcp 145.79.227.11:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.396744433 (296h45m3.993515466s) [noise=true , address=145.79.227.2:4115 , piece_id=P3OEN6MLRVOH5JTV52QPQ2QQNTLVAK46XU2RTPQ2TAWZ3SEFFQEQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.398824473 (296h45m3.995595505s) [noise=true , address=145.79.227.51:4414 , piece_id=7ZSATDNULGY5IM6V4IU7H6CC7FX3Q7NZFTSHNXINCR2ZL2JRZWXA, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:1Gh17XmEkfiFPqqR1ASnC7LoMz6ik3a5EHitxG67GVQkDxAHc2, address:145.79.227.51:4414): protocol: rpc: dial tcp 145.79.227.51:4414: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.399193187 (296h45m3.995964222s) [noise=true , address=145.79.227.41:4087 , piece_id=NS4H2CPABLOBYU54TNIDFNJLKKKVCHR43EYBBBN3KWUAFL34BL5A, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.399733848 (296h45m3.99650488s) [noise=true , address=145.79.227.5:4224 , piece_id=DXXZ7MSO75W5GUBBERBFQCERX6JO6ODUCOK7KKUNFX2EWI5WZMYQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:145.79.227.5:4224): protocol: rpc: dial tcp 145.79.227.5:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.401539291 (296h45m3.998310327s) [noise=true , address=145.79.227.14:4839 , piece_id=4LFKQFFLBS2B4GANRSGF2WDFERCYEWVL3U3SKUHTNR4U3CSFF3BQ, seg_pos={0 0}, upload=332377]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:45:06.401587421 (296h45m3.998358433s) [seg_pos={0 0}, upload=332377]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 19:45:06.428100280 (296h45m4.024871314s) [upload=332377]: Done uploading segments... Mar 18 19:45:06.428141182 (296h45m4.024912253s) [upload=332377]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.055280882s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XXXXXX.XX....] duration: 1.480770978s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:44:46 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.480818771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 1.561475702s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:44:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.561505796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.........................................XXXXXXXXX.XXX...XXX] duration: 5.002441336s trace: https://jaeger.dev.storj.io/trace/6cf77f86ea54c70a FAIL (2026-03-18 19:44:41 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002465574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.07639423s trace: https://jaeger.dev.storj.io/trace/56b8d22a754b5364 FAIL (2026-03-18 19:44:30 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 32 (296h43m58.190450361s) [seg_pos={0 0}, upload=332365]: Done begin upload segment. Mar 18 19:44:00.593745658 (296h43m58.190516692s) [noise=true , address=145.79.227.12:4770 , piece_id=USJOP2V7CXPEM7BL4BEXYGVJZEO2DTB5H5EH4BCX23YEKXHSFM7Q, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.593800277 (296h43m58.190571308s) [noise=true , address=145.79.227.21:4479 , piece_id=ZVRWYCQ6CGNFJ73ZJED2DGHUBE4XFMCD7TS4MWOANWRBWEIDOWEQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.594487300 (296h43m58.191258335s) [noise=true , address=145.79.227.39:4017 , piece_id=FEGRXQLT5U2OPSDDD5J565A3TCWFJ2ACA4UDG6SH6UXDPTD2OEWA, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.595513197 (296h43m58.192284229s) [noise=true , address=145.79.227.25:4623 , piece_id=NMHX7L4CSFXIK2NV4VEBLDLNTPYB5LN2ACHSKCWTR5Q6SC7RHSZQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.596223175 (296h43m58.19299421s) [noise=true , address=145.79.227.51:4413 , piece_id=M3VO55VRCKRCIP5DWXO7SXCHKWE6RA36YENVM7QIV3LZ6UETDBDA, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.596907659 (296h43m58.193678693s) [noise=true , address=145.79.227.49:4338 , piece_id=GARCPQP2YC4XQEMUW5HOCAVMUYLBN4EUE77PPNR3QHELQLFOOREQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.597562343 (296h43m58.194333371s) [noise=true , address=45.91.116.14:4196 , piece_id=FQI252JXOEP3JJY56SY5TKSYL6Y43A6LRQOKPCICNPRTZP6FJXUA, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.598133058 (296h43m58.194904089s) [noise=true , address=77.90.31.13:4629 , piece_id=JGFQ5DY4S3AFJYQQANASGSE4HX76DQBLC7KZYEGLZHA2KHN4NUXQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.598633619 (296h43m58.195404644s) [noise=true , address=45.91.116.5:4297 , piece_id=PRN3XP5YR5PDZY6RZXGDUXGLI6SJGGRUAUUIUWJ5564HOUSKAKAQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.599190137 (296h43m58.195961167s) [noise=true , address=79.127.219.38:4821 , piece_id=R244X2PZ4JKHDIDEKDVZ3ZI7XZMNLV52P6QXEFUNH7YR4DUEI4AQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.599213666 (296h43m58.195984699s) [noise=true , address=145.79.227.43:4158 , piece_id=EM2V5CPLDMKMIQLXMOD6XC2COTSEDGAZBTA5DKXA63R6XUS5Q2DA, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.599776948 (296h43m58.196547976s) [noise=true , address=77.90.31.21:4877 , piece_id=QXT46PQMCPEBHIQBZJ4MP5AYOA55SWIWHPCMJFYBNITKDEIZNCGA, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.600261191 (296h43m58.197032217s) [noise=true , address=145.79.227.31:4802 , piece_id=6DPSJ25JOKJ5FJVOOS5CP2FS2SJAK7LGADXLX2P3HGNNJQEHDP2A, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.602528448 (296h43m58.199299493s) [noise=true , address=145.79.227.29:4946 , piece_id=E5PIJAKAZGQRH63QYWWWNTC623HXKZIR4NMLO7AJDU6XKLVD6KHA, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.604716671 (296h43m58.201487704s) [noise=true , address=145.79.227.52:4446 , piece_id=W7UJEVPE2H3QI5FYWD7MZUK6ZMFJH6SWVFBPLPCRFOH4DHUDTBPQ, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.607857736 (296h43m58.204628765s) [noise=true , address=145.79.227.22:4514 , piece_id=G5AHGRDWQYUHLDYLWPOHETJYDKV2HTO4XMZSR6XNGH4T2O2I3W2Q, seg_pos={0 0}, upload=332365]: Uploading piece... Mar 18 19:44:00.676386953 (296h43m58.273157976s) [upload=332365]: Next returned nil segment Mar 18 19:44:00.676402622 (296h43m58.273173649s) [upload=332365]: Waiting for error group managing segments... Mar 18 19:44:00.680112422 (296h43m58.276883436s) [noise=true , address=45.91.116.14:4196 , piece_id=FQI252JXOEP3JJY56SY5TKSYL6Y43A6LRQOKPCICNPRTZP6FJXUA, seg_pos={0 0}, upload=332365]: Done uploading piece. err: <nil> Mar 18 19:44:00.680372117 (296h43m58.277143151s) [noise=true , address=45.91.116.5:4297 , piece_id=PRN3XP5YR5PDZY6RZXGDUXGLI6SJGGRUAUUIUWJ5564HOUSKAKAQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: <nil> Mar 18 19:44:00.680430094 (296h43m58.277201117s) [noise=true , address=77.90.31.21:4877 , piece_id=QXT46PQMCPEBHIQBZJ4MP5AYOA55SWIWHPCMJFYBNITKDEIZNCGA, seg_pos={0 0}, upload=332365]: Done uploading piece. err: <nil> Mar 18 19:44:00.693803090 (296h43m58.290574115s) [noise=true , address=77.90.31.13:4629 , piece_id=JGFQ5DY4S3AFJYQQANASGSE4HX76DQBLC7KZYEGLZHA2KHN4NUXQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: <nil> Mar 18 19:44:00.700564788 (296h43m58.297335801s) [noise=true , address=79.127.219.38:4821 , piece_id=R244X2PZ4JKHDIDEKDVZ3ZI7XZMNLV52P6QXEFUNH7YR4DUEI4AQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: <nil> Mar 18 19:44:30.504947693 (296h44m28.101718719s) [noise=true , address=145.79.227.51:4413 , piece_id=M3VO55VRCKRCIP5DWXO7SXCHKWE6RA36YENVM7QIV3LZ6UETDBDA, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1oMVt1KZoNSDX5jKHBXw4WSr69eQYxcY3Z1nxLBsHr19Btw7Gy, address:145.79.227.51:4413): protocol: rpc: dial tcp 145.79.227.51:4413: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.504976626 (296h44m28.101747657s) [noise=true , address=145.79.227.39:4017 , piece_id=FEGRXQLT5U2OPSDDD5J565A3TCWFJ2ACA4UDG6SH6UXDPTD2OEWA, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk, address:145.79.227.39:4017): protocol: rpc: dial tcp 145.79.227.39:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.505520099 (296h44m28.102291123s) [noise=true , address=145.79.227.29:4946 , piece_id=E5PIJAKAZGQRH63QYWWWNTC623HXKZIR4NMLO7AJDU6XKLVD6KHA, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX, address:145.79.227.29:4946): protocol: rpc: dial tcp 145.79.227.29:4946: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.505762213 (296h44m28.102533243s) [noise=true , address=145.79.227.43:4158 , piece_id=EM2V5CPLDMKMIQLXMOD6XC2COTSEDGAZBTA5DKXA63R6XUS5Q2DA, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.505822371 (296h44m28.102593401s) [noise=true , address=145.79.227.49:4338 , piece_id=GARCPQP2YC4XQEMUW5HOCAVMUYLBN4EUE77PPNR3QHELQLFOOREQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: rpc: dial tcp 145.79.227.49:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.505981268 (296h44m28.102752302s) [noise=true , address=145.79.227.22:4514 , piece_id=G5AHGRDWQYUHLDYLWPOHETJYDKV2HTO4XMZSR6XNGH4T2O2I3W2Q, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:145.79.227.22:4514): protocol: rpc: dial tcp 145.79.227.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.506095008 (296h44m28.102866037s) [noise=true , address=145.79.227.52:4446 , piece_id=W7UJEVPE2H3QI5FYWD7MZUK6ZMFJH6SWVFBPLPCRFOH4DHUDTBPQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:145.79.227.52:4446): protocol: rpc: dial tcp 145.79.227.52:4446: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.506306341 (296h44m28.10307737s) [noise=true , address=145.79.227.21:4479 , piece_id=ZVRWYCQ6CGNFJ73ZJED2DGHUBE4XFMCD7TS4MWOANWRBWEIDOWEQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:12VCopFEoHk99GsQNQPuWvTQZRD1bpXi8HHLKXaXRdvFThTxmAk, address:145.79.227.21:4479): protocol: rpc: dial tcp 145.79.227.21:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.506478948 (296h44m28.103249981s) [noise=true , address=145.79.227.31:4802 , piece_id=6DPSJ25JOKJ5FJVOOS5CP2FS2SJAK7LGADXLX2P3HGNNJQEHDP2A, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1iMEuZv8uHpYjTqP9JVpznKPPdEyyy4xMTsgVUGMuCjZkV4cri, address:145.79.227.31:4802): protocol: rpc: dial tcp 145.79.227.31:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.506572237 (296h44m28.103343266s) [noise=true , address=145.79.227.12:4770 , piece_id=USJOP2V7CXPEM7BL4BEXYGVJZEO2DTB5H5EH4BCX23YEKXHSFM7Q, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.506894692 (296h44m28.103665727s) [noise=true , address=145.79.227.25:4623 , piece_id=NMHX7L4CSFXIK2NV4VEBLDLNTPYB5LN2ACHSKCWTR5Q6SC7RHSZQ, seg_pos={0 0}, upload=332365]: Done uploading piece. err: ecclient: upload failed (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5, address:145.79.227.25:4623): protocol: rpc: dial tcp 145.79.227.25:4623: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:44:30.506949648 (296h44m28.103720676s) [seg_pos={0 0}, upload=332365]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Mar 18 19:44:30.579296096 (296h44m28.17606712s) [upload=332365]: Done uploading segments... Mar 18 19:44:30.579333401 (296h44m28.176104424s) [upload=332365]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.076472848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.049224461s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.049265553s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 1.696977297s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:44:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.697011513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 24.106202975s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:44:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.106237606s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..........................................XXXXXXXXX.XXX...XX] duration: 5.002164979s trace: https://jaeger.dev.storj.io/trace/215219882e97608f FAIL (2026-03-18 19:44:09 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002190877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 4.419025613s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.419055924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [................................................XXXXXX.XX...] duration: 1.088178782s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:43:47 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.088210764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.0322442s trace: https://jaeger.dev.storj.io/trace/31f922f6ea789256 FAIL (2026-03-18 19:43:25 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 19:42:55.681658763 (296h42m53.278429796s) [upload=332345]: Starting upload Mar 18 19:42:55.681802318 (296h42m53.278573351s) [upload=332345]: Done starting upload Mar 18 19:42:55.681912234 (296h42m53.278683247s) [upload=332345]: Uploading segments... Mar 18 19:42:55.682308388 (296h42m53.279079424s) [upload=332345]: Got next segment. Inline: false Mar 18 19:42:55.682411937 (296h42m53.279182971s) [upload=332345]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:42:55.682419312 (296h42m53.279190345s) [upload=332345]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:42:56.043789020 (296h42m53.640560052s) [seg_pos={0 0}, upload=332345]: Begin upload segment... Mar 18 19:42:56.043860263 (296h42m53.640631298s) [seg_pos={0 0}, upload=332345]: Done begin upload segment. Mar 18 19:42:56.043935663 (296h42m53.640706696s) [noise=true , address=45.91.116.13:4155 , piece_id=Z373HRGM3DJOQXUV5XJCFOVR2TRZDUNWVXPKTSF5JKCFIE354ZYQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.044000364 (296h42m53.640771398s) [noise=true , address=79.127.219.41:4928 , piece_id=RLRTSLL7246XZHHHR3HG43Z2UYIMH2TJTE5KPFU726PVTYKO2BGQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.044086767 (296h42m53.640857804s) [noise=true , address=145.79.227.5:4221 , piece_id=YH7LCWIRXZX56CZHFZYKYF7HPD2Y7HNHX2P3X7H7I2Y3UMSAHFDA, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.044736056 (296h42m53.641507134s) [noise=true , address=45.91.116.12:4244 , piece_id=3RELV4VLHQS53WEAUOEQB6NRR72SYGGWXC5YEF3UDZKXQIEKRUBQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.044841275 (296h42m53.641612309s) [noise=true , address=145.79.227.33:4871 , piece_id=BIESU7J5IS7BEBDHVAQ5PGPYMUBUYTTEO3OK5X7TMGVEXQPTNDUA, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.045494921 (296h42m53.642265956s) [noise=true , address=77.90.31.18:4805 , piece_id=RRJWTYJ2J7G5GHIEXF3AVEETJIIBTZRD43X4T5FTQ7KSXY4GYL6A, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.046695475 (296h42m53.643466502s) [noise=true , address=79.127.219.38:4819 , piece_id=6UHSJANQOXUBYOK5J2M3R3EGZFMN7SKTBVG2GCPXG3S54PRHZLFA, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.048163653 (296h42m53.644934689s) [noise=true , address=145.79.227.30:4765 , piece_id=ON3HXHXJGEZRUFDJYF2WATJWOMM35QOG5A77LLZARGORJBWGHLGA, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.049808355 (296h42m53.646579386s) [noise=true , address=77.90.31.13:4630 , piece_id=ZZVIREUOHVBUWEWEKYXK5ZWGO2TM4RDYFEFFXYLRWDCWGBFKILIQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.051574181 (296h42m53.648345215s) [noise=true , address=45.91.116.6:4974 , piece_id=6YBECRFDMAUTPOWZRFKPVBF6IWFKCURFKIR5TBVXD2AV4F2Y5XDQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.052480343 (296h42m53.649251425s) [noise=true , address=45.91.116.11:4121 , piece_id=JNMT24C5BZ73YB5WHAQ5QAPERWNZ4NLQIZ7CRLFGSTQDGXHFGZPQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.053170259 (296h42m53.649941284s) [noise=true , address=145.79.227.7:4298 , piece_id=A4QFHO6AORTUF5PDJJJTFS2MQXCDCUJ4ADY7Q4NF4YLYXRWYKERQ, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.053980992 (296h42m53.650752016s) [noise=true , address=77.90.31.15:4700 , piece_id=NAOA3IXX6NSUBJLZSCE3AUC7L4A645YQX2C4HYF6YEVRRK5CAZ3A, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.054706305 (296h42m53.651477317s) [noise=true , address=45.91.116.7:4019 , piece_id=APJ4BALLFIMJSMZL4LDDLNCKAWIC7MHRLBF7UF526C2D7DPFVQDA, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.055315122 (296h42m53.652086148s) [noise=true , address=145.79.227.20:4442 , piece_id=BQU6477KQFWIMV4FRQV6NTUPDSFCLZBW5AWSVFYZ2RIHRPRIAQ3A, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.056084347 (296h42m53.65285538s) [noise=true , address=145.79.227.29:4942 , piece_id=IUNK4BWBGZ7HZMZJUB6L7TCAW6HNXNKT36MD424SIZA5GNCXHB2Q, seg_pos={0 0}, upload=332345]: Uploading piece... Mar 18 19:42:56.122307551 (296h42m53.719078585s) [upload=332345]: Next returned nil segment Mar 18 19:42:56.122325073 (296h42m53.719096106s) [upload=332345]: Waiting for error group managing segments... Mar 18 19:42:56.127748496 (296h42m53.724519532s) [noise=true , address=45.91.116.7:4019 , piece_id=APJ4BALLFIMJSMZL4LDDLNCKAWIC7MHRLBF7UF526C2D7DPFVQDA, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.128899504 (296h42m53.725670532s) [noise=true , address=77.90.31.15:4700 , piece_id=NAOA3IXX6NSUBJLZSCE3AUC7L4A645YQX2C4HYF6YEVRRK5CAZ3A, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.129217140 (296h42m53.725988155s) [noise=true , address=77.90.31.13:4630 , piece_id=ZZVIREUOHVBUWEWEKYXK5ZWGO2TM4RDYFEFFXYLRWDCWGBFKILIQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.143213584 (296h42m53.739984618s) [noise=true , address=45.91.116.11:4121 , piece_id=JNMT24C5BZ73YB5WHAQ5QAPERWNZ4NLQIZ7CRLFGSTQDGXHFGZPQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.143886690 (296h42m53.740657727s) [noise=true , address=45.91.116.13:4155 , piece_id=Z373HRGM3DJOQXUV5XJCFOVR2TRZDUNWVXPKTSF5JKCFIE354ZYQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.144234704 (296h42m53.741005732s) [noise=true , address=79.127.219.38:4819 , piece_id=6UHSJANQOXUBYOK5J2M3R3EGZFMN7SKTBVG2GCPXG3S54PRHZLFA, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.151626423 (296h42m53.748397453s) [noise=true , address=45.91.116.12:4244 , piece_id=3RELV4VLHQS53WEAUOEQB6NRR72SYGGWXC5YEF3UDZKXQIEKRUBQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.157595893 (296h42m53.754366925s) [noise=true , address=45.91.116.6:4974 , piece_id=6YBECRFDMAUTPOWZRFKPVBF6IWFKCURFKIR5TBVXD2AV4F2Y5XDQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.162359372 (296h42m53.759130383s) [noise=true , address=77.90.31.18:4805 , piece_id=RRJWTYJ2J7G5GHIEXF3AVEETJIIBTZRD43X4T5FTQ7KSXY4GYL6A, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:42:56.168139938 (296h42m53.764910958s) [noise=true , address=79.127.219.41:4928 , piece_id=RLRTSLL7246XZHHHR3HG43Z2UYIMH2TJTE5KPFU726PVTYKO2BGQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: <nil> Mar 18 19:43:25.397790108 (296h43m22.994561129s) [noise=true , address=145.79.227.30:4765 , piece_id=ON3HXHXJGEZRUFDJYF2WATJWOMM35QOG5A77LLZARGORJBWGHLGA, seg_pos={0 0}, upload=332345]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: rpc: dial tcp 145.79.227.30:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:43:25.397876868 (296h43m22.9946479s) [noise=true , address=145.79.227.29:4942 , piece_id=IUNK4BWBGZ7HZMZJUB6L7TCAW6HNXNKT36MD424SIZA5GNCXHB2Q, seg_pos={0 0}, upload=332345]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:145.79.227.29:4942): protocol: rpc: dial tcp 145.79.227.29:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:43:25.398436518 (296h43m22.995207552s) [noise=true , address=145.79.227.5:4221 , piece_id=YH7LCWIRXZX56CZHFZYKYF7HPD2Y7HNHX2P3X7H7I2Y3UMSAHFDA, seg_pos={0 0}, upload=332345]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:43:25.398462882 (296h43m22.995233914s) [noise=true , address=145.79.227.7:4298 , piece_id=A4QFHO6AORTUF5PDJJJTFS2MQXCDCUJ4ADY7Q4NF4YLYXRWYKERQ, seg_pos={0 0}, upload=332345]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:43:25.398626084 (296h43m22.99539711s) [noise=true , address=145.79.227.33:4871 , piece_id=BIESU7J5IS7BEBDHVAQ5PGPYMUBUYTTEO3OK5X7TMGVEXQPTNDUA, seg_pos={0 0}, upload=332345]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:145.79.227.33:4871): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:43:25.398898186 (296h43m22.995669221s) [noise=true , address=145.79.227.20:4442 , piece_id=BQU6477KQFWIMV4FRQV6NTUPDSFCLZBW5AWSVFYZ2RIHRPRIAQ3A, seg_pos={0 0}, upload=332345]: Done uploading piece. err: ecclient: upload failed (node:125KyoqeSzbjP8XPhXqw2AenTDiLtVmt5KpLBD7ZW7YC9e9zJu9, address:145.79.227.20:4442): protocol: rpc: dial tcp 145.79.227.20:4442: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:43:25.399270541 (296h43m22.996041566s) [seg_pos={0 0}, upload=332345]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 19:43:25.428060567 (296h43m23.024831605s) [upload=332345]: Done uploading segments... Mar 18 19:43:25.428104244 (296h43m23.024875281s) [upload=332345]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.032280388s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...........................................XXXXXXXXX.XXX...X] duration: 5.001447299s trace: https://jaeger.dev.storj.io/trace/4758c2682b0e26a FAIL (2026-03-18 19:43:18 UTC) operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001470436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration: 3.899042146s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:42:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.899074366s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [............................................XXXXXXXXX.XXX...] duration: 1.748643118s trace: https://jaeger.dev.storj.io/trace/6e68742e8267acf6 FAIL (2026-03-18 19:42:43 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.74866738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.274196025s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:42:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.274231432s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.................................................XXXXXX.XX..] duration: 1.628579612s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:42:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.62860919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.069365895s trace: https://jaeger.dev.storj.io/trace/14b317da90285717 FAIL (2026-03-18 19:42:11 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 19:41:41.564577253 (296h41m39.161348276s) [upload=332325]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:41:41.644122396 (296h41m39.24089342s) [seg_pos={0 0}, upload=332325]: Begin upload segment... Mar 18 19:41:41.644226443 (296h41m39.240997468s) [seg_pos={0 0}, upload=332325]: Done begin upload segment. Mar 18 19:41:41.644317514 (296h41m39.241088558s) [noise=true , address=145.79.227.44:4198 , piece_id=XWV6MTQAF3XCZ3SEYJO4APL6UZ7XGMJSQCGWDZPTWBN3K7AMJLEQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.645690290 (296h41m39.242461324s) [noise=true , address=145.79.227.20:4443 , piece_id=LDMULMPXWAJKEKHYMB7KCEWJTEVJDOVLDJDWLFS5CYNHWO2CXE3Q, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.645707357 (296h41m39.242478387s) [noise=true , address=77.90.31.13:4626 , piece_id=AHJYRXLWP7R6W575ZZYW3CKZ5XC55FMS5KAVDQ6FFPKND2INOGGQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.646227598 (296h41m39.242998628s) [noise=true , address=145.79.227.47:4270 , piece_id=LPUTEFKDUJB6YFH6HPVV5P5KXRAEVHTROIUMOMGRQ3QQYBJI4MEQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.646297701 (296h41m39.243068724s) [noise=true , address=145.79.227.48:4302 , piece_id=57OTCD2LBMAYVGVYIJLT6ZVQ66IL7AR2VSIOGRHDIH57I7Q2C3OA, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.646689693 (296h41m39.243460726s) [noise=true , address=77.90.31.21:4877 , piece_id=XUW2BONFSD6VYPDUGQO7N2DISLM6S7UIODTBNVADWHCEQTEIEPTQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.647472383 (296h41m39.244243427s) [noise=true , address=77.90.31.16:4736 , piece_id=JRZH5SSGPBLWV2AZCJW5PRI3FMQ57NVSAPJRYY7RY6VPNWHXWMFQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.647738721 (296h41m39.244509754s) [noise=true , address=145.79.227.15:4878 , piece_id=N7B24FIFITAZYDKK765PDXZVVWKNYEAOKHMWNAJAUQBBJD7PVUWQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.648038544 (296h41m39.244809577s) [noise=true , address=45.91.116.12:4260 , piece_id=RW3XPRU3T45Q4YXN66AHQELJMY2XSJS73EGNTFXPEICO3Z4KLKIA, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.648996015 (296h41m39.245767043s) [noise=true , address=145.79.227.9:4371 , piece_id=YTIDFGJCJBEMDJHMQXGC363LFJT3CEJUJHACDMO5PD2SRIDPX4GQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.649689782 (296h41m39.246460821s) [noise=true , address=145.79.227.50:4373 , piece_id=74VQRON57XGBDSFZGXO2MRWXXJUN3JSEJ2YRM7O7WYE7KTDQ7Q3Q, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.650639477 (296h41m39.247410511s) [noise=true , address=145.79.227.31:4803 , piece_id=7BIPZVEHUKIS5MIFOFIUEAXB4NWOKNGPY4I5E6XQQCJDXZN7C44Q, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.651985195 (296h41m39.248756221s) [noise=true , address=145.79.227.34:4909 , piece_id=CVV57EW7FBZ6UANSKCWLNURVLZLIZ6HCKDQKDUSEZBF67ENGBGBQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.652798424 (296h41m39.24956945s) [noise=true , address=145.79.227.11:4734 , piece_id=ZVOJOJFM4CG5IELFAMPJQC4Q55FCVUQBEY6FUPWDAEYDA4NX7K2A, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.654221424 (296h41m39.250992457s) [noise=true , address=77.90.31.11:4556 , piece_id=MSZTZFZD2E7L6L4AC6Q6MHRQUCABHDIG2RLD6EGC3I3S2JWACFNQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.655066065 (296h41m39.251837098s) [noise=true , address=45.91.116.7:4012 , piece_id=VKHQZRZ2PWFDC6CVH7LYRN26X5Y5ZFMSL6OIMWFICWA5ZHNFPWTQ, seg_pos={0 0}, upload=332325]: Uploading piece... Mar 18 19:41:41.733123459 (296h41m39.329894487s) [upload=332325]: Next returned nil segment Mar 18 19:41:41.733140738 (296h41m39.329911768s) [upload=332325]: Waiting for error group managing segments... Mar 18 19:41:41.737621397 (296h41m39.334392425s) [noise=true , address=45.91.116.7:4012 , piece_id=VKHQZRZ2PWFDC6CVH7LYRN26X5Y5ZFMSL6OIMWFICWA5ZHNFPWTQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: <nil> Mar 18 19:41:41.737650690 (296h41m39.334421718s) [noise=true , address=45.91.116.12:4260 , piece_id=RW3XPRU3T45Q4YXN66AHQELJMY2XSJS73EGNTFXPEICO3Z4KLKIA, seg_pos={0 0}, upload=332325]: Done uploading piece. err: <nil> Mar 18 19:41:41.738254570 (296h41m39.335025597s) [noise=true , address=77.90.31.21:4877 , piece_id=XUW2BONFSD6VYPDUGQO7N2DISLM6S7UIODTBNVADWHCEQTEIEPTQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: <nil> Mar 18 19:41:41.740273139 (296h41m39.337044168s) [noise=true , address=77.90.31.16:4736 , piece_id=JRZH5SSGPBLWV2AZCJW5PRI3FMQ57NVSAPJRYY7RY6VPNWHXWMFQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: <nil> Mar 18 19:41:41.752602429 (296h41m39.349373453s) [noise=true , address=77.90.31.11:4556 , piece_id=MSZTZFZD2E7L6L4AC6Q6MHRQUCABHDIG2RLD6EGC3I3S2JWACFNQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: <nil> Mar 18 19:41:41.758093684 (296h41m39.354864696s) [noise=true , address=77.90.31.13:4626 , piece_id=AHJYRXLWP7R6W575ZZYW3CKZ5XC55FMS5KAVDQ6FFPKND2INOGGQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: <nil> Mar 18 19:42:11.519593765 (296h42m9.116364795s) [noise=true , address=145.79.227.15:4878 , piece_id=N7B24FIFITAZYDKK765PDXZVVWKNYEAOKHMWNAJAUQBBJD7PVUWQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC, address:145.79.227.15:4878): protocol: rpc: dial tcp 145.79.227.15:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.520192396 (296h42m9.116963434s) [noise=true , address=145.79.227.34:4909 , piece_id=CVV57EW7FBZ6UANSKCWLNURVLZLIZ6HCKDQKDUSEZBF67ENGBGBQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh, address:145.79.227.34:4909): protocol: rpc: dial tcp 145.79.227.34:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.520313781 (296h42m9.117084815s) [noise=true , address=145.79.227.11:4734 , piece_id=ZVOJOJFM4CG5IELFAMPJQC4Q55FCVUQBEY6FUPWDAEYDA4NX7K2A, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:145.79.227.11:4734): protocol: rpc: dial tcp 145.79.227.11:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.520721436 (296h42m9.11749247s) [noise=true , address=145.79.227.20:4443 , piece_id=LDMULMPXWAJKEKHYMB7KCEWJTEVJDOVLDJDWLFS5CYNHWO2CXE3Q, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:1rTVUAPUj4PUy8qHuutnB2WYCVpJdiVwfutA6UK4TZoPzvQEUv, address:145.79.227.20:4443): protocol: rpc: dial tcp 145.79.227.20:4443: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.520755590 (296h42m9.117526625s) [noise=true , address=145.79.227.47:4270 , piece_id=LPUTEFKDUJB6YFH6HPVV5P5KXRAEVHTROIUMOMGRQ3QQYBJI4MEQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:12sJfMr1nMTpizd5PZ2bZ6M8tZWhFL2ukdJ3cNSRQtdUHNJhg81, address:145.79.227.47:4270): protocol: rpc: dial tcp 145.79.227.47:4270: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.521051805 (296h42m9.11782284s) [noise=true , address=145.79.227.9:4371 , piece_id=YTIDFGJCJBEMDJHMQXGC363LFJT3CEJUJHACDMO5PD2SRIDPX4GQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:145.79.227.9:4371): protocol: rpc: dial tcp 145.79.227.9:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.521190057 (296h42m9.117961091s) [noise=true , address=145.79.227.31:4803 , piece_id=7BIPZVEHUKIS5MIFOFIUEAXB4NWOKNGPY4I5E6XQQCJDXZN7C44Q, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:145.79.227.31:4803): protocol: rpc: dial tcp 145.79.227.31:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.521369520 (296h42m9.118140554s) [noise=true , address=145.79.227.50:4373 , piece_id=74VQRON57XGBDSFZGXO2MRWXXJUN3JSEJ2YRM7O7WYE7KTDQ7Q3Q, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: rpc: dial tcp 145.79.227.50:4373: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.521672210 (296h42m9.118443244s) [noise=true , address=145.79.227.44:4198 , piece_id=XWV6MTQAF3XCZ3SEYJO4APL6UZ7XGMJSQCGWDZPTWBN3K7AMJLEQ, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:145.79.227.44:4198): protocol: rpc: dial tcp 145.79.227.44:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.522269607 (296h42m9.11904064s) [noise=true , address=145.79.227.48:4302 , piece_id=57OTCD2LBMAYVGVYIJLT6ZVQ66IL7AR2VSIOGRHDIH57I7Q2C3OA, seg_pos={0 0}, upload=332325]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:145.79.227.48:4302): protocol: rpc: dial tcp 145.79.227.48:4302: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:42:11.522330524 (296h42m9.119101543s) [seg_pos={0 0}, upload=332325]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Mar 18 19:42:11.586406381 (296h42m9.183177413s) [upload=332325]: Done uploading segments... Mar 18 19:42:11.586448490 (296h42m9.183219519s) [upload=332325]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.069391804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.666216113s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.666250889s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XXXXXX.XX.] duration: 1.302272863s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:42:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.302300464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XXXXXX.XX] duration: 1m0.001208431s trace: https://jaeger.dev.storj.io/trace/4b4d10a8051ad01b FAIL (2026-03-18 19:42:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001226317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XXX......] duration: 1.687564906s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:41:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.68759611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.............................................XXXXXXXXX.XXX..] duration: 921.626746ms trace: https://jaeger.dev.storj.io/trace/6e68742e8267acf6 FAIL (2026-03-18 19:41:53 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 921.720631ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.327675531s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:41:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.327786438s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXXX.X] duration: 1m0.000811561s trace: https://jaeger.dev.storj.io/trace/22bdd442a230bdd2 FAIL (2026-03-18 19:41:08 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000837272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.031333362s trace: https://jaeger.dev.storj.io/trace/675bdc8178a75cb4 FAIL (2026-03-18 19:41:03 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== pload=332303]: Done starting upload Mar 18 19:40:33.571892650 (296h40m31.168663674s) [upload=332303]: Uploading segments... Mar 18 19:40:33.572305045 (296h40m31.16907606s) [upload=332303]: Got next segment. Inline: false Mar 18 19:40:33.572377523 (296h40m31.169148552s) [upload=332303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:40:33.572386058 (296h40m31.169157081s) [upload=332303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:40:33.608244760 (296h40m31.205015792s) [seg_pos={0 0}, upload=332303]: Begin upload segment... Mar 18 19:40:33.608337184 (296h40m31.205108216s) [seg_pos={0 0}, upload=332303]: Done begin upload segment. Mar 18 19:40:33.608403405 (296h40m31.205174441s) [noise=true , address=145.79.227.41:4085 , piece_id=FZPDM6GDXMAOWSGCNPU33GEKWICPWNOE3XNYFAOWZ3AFXAUXW4VA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.609718697 (296h40m31.20648973s) [noise=true , address=145.79.227.39:4017 , piece_id=UAN774WCTOGV5LMWAVQ2J7STHC2EP2OCH3U5AH6FP6EDSHHURFKA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.610442266 (296h40m31.2072133s) [noise=true , address=77.90.31.21:4877 , piece_id=3DBEPHOBZBPTUJN44YPFEYCVMNPIS7VKVHTOIRRFPHU4BJWKZEYA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.611164527 (296h40m31.20793556s) [noise=true , address=45.91.116.11:4152 , piece_id=TQNVL5CHWZUGC5MCROL4HKVY7CJCZ7FK2CSUK5XT7IGGCJ4KIQJQ, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.611212241 (296h40m31.207983274s) [noise=true , address=77.90.31.14:4664 , piece_id=2WTVFWMVPRN6REGU7GSMSQFB4W436ADXBO7Q54PG64WDANJWYVPA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.611879791 (296h40m31.208650825s) [noise=true , address=145.79.227.8:4335 , piece_id=RMFE4RQFSYJENGNW77Y5CFDLEKQKR6DFHBWQZBMIL323YBYVWJSA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.612187772 (296h40m31.208958802s) [noise=true , address=79.127.219.41:4927 , piece_id=NGION5UIFUPTSLH4GIMQ2PMU6755ZIXVFG3PUTWMDUHCKY6664ZQ, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.612517683 (296h40m31.209288711s) [noise=true , address=145.79.227.42:4126 , piece_id=VY2L2VREHCXEUIGES5B4XF7XMIHBFXT72XPF37LG745GPJOL3EZA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.612855515 (296h40m31.209626539s) [noise=true , address=145.79.227.20:4441 , piece_id=XHY6L42WGOOQ3ZOB4NG3LD6XK4EHDV4VWNCCDNNPBBXZY3UK5I6A, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.613501854 (296h40m31.21027288s) [noise=true , address=77.90.31.13:4630 , piece_id=LKQHF5EPVOB7ON6BEP2Y2NNTIF2SQ5I2OTOEMH25ZMLPXQLE4E4A, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.613512351 (296h40m31.210283374s) [noise=true , address=145.79.227.2:4113 , piece_id=4K64EHXXSVEXOGM6CZ3IANE7P5BVH2T4NOFAYMKBFDCOCCIXMPYQ, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.614203534 (296h40m31.210974566s) [noise=true , address=45.91.116.14:4191 , piece_id=BZWQAOZ242CHGTD3U3MKVO7BNR2DCDFUZ7G3LEQWLRR27Q2LYGRA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.614366827 (296h40m31.21113786s) [noise=true , address=77.90.31.11:4557 , piece_id=32SGJYXTTDUCUOMIM6XLM3VRC6T2HO3C3WUR4MXV3ATEGB3GSNKA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.615110723 (296h40m31.211881756s) [noise=true , address=145.79.227.43:4158 , piece_id=WPAXDH6WSWAOICNIGVJHEKGM2SGHZYHIVWDEKOXEQSCLX5UOTZVQ, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.615238981 (296h40m31.212010009s) [noise=true , address=145.79.227.12:4770 , piece_id=UHOUUDH4JGQWJWMRR7MQIDTF4TT5X7FTACQO7SXXK3KE5RQPDQEQ, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.616247409 (296h40m31.213018436s) [noise=true , address=145.79.227.48:4304 , piece_id=CFYFXRX2L44XYVASI66OUKPJNGS67LD67HU4MZ73GP6NL6ORF3OA, seg_pos={0 0}, upload=332303]: Uploading piece... Mar 18 19:40:33.688536071 (296h40m31.285307096s) [upload=332303]: Next returned nil segment Mar 18 19:40:33.688572892 (296h40m31.285343925s) [upload=332303]: Waiting for error group managing segments... Mar 18 19:40:33.691403615 (296h40m31.288174629s) [noise=true , address=77.90.31.21:4877 , piece_id=3DBEPHOBZBPTUJN44YPFEYCVMNPIS7VKVHTOIRRFPHU4BJWKZEYA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:40:33.691719186 (296h40m31.288490209s) [noise=true , address=45.91.116.11:4152 , piece_id=TQNVL5CHWZUGC5MCROL4HKVY7CJCZ7FK2CSUK5XT7IGGCJ4KIQJQ, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:40:33.692198751 (296h40m31.288969778s) [noise=true , address=79.127.219.41:4927 , piece_id=NGION5UIFUPTSLH4GIMQ2PMU6755ZIXVFG3PUTWMDUHCKY6664ZQ, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:40:33.692346530 (296h40m31.289117568s) [noise=true , address=77.90.31.11:4557 , piece_id=32SGJYXTTDUCUOMIM6XLM3VRC6T2HO3C3WUR4MXV3ATEGB3GSNKA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:40:33.692949260 (296h40m31.289720293s) [noise=true , address=77.90.31.14:4664 , piece_id=2WTVFWMVPRN6REGU7GSMSQFB4W436ADXBO7Q54PG64WDANJWYVPA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:40:33.702534002 (296h40m31.299305035s) [noise=true , address=45.91.116.14:4191 , piece_id=BZWQAOZ242CHGTD3U3MKVO7BNR2DCDFUZ7G3LEQWLRR27Q2LYGRA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:40:33.728685016 (296h40m31.325456048s) [noise=true , address=77.90.31.13:4630 , piece_id=LKQHF5EPVOB7ON6BEP2Y2NNTIF2SQ5I2OTOEMH25ZMLPXQLE4E4A, seg_pos={0 0}, upload=332303]: Done uploading piece. err: <nil> Mar 18 19:41:03.534213420 (296h41m1.130984454s) [noise=true , address=145.79.227.41:4085 , piece_id=FZPDM6GDXMAOWSGCNPU33GEKWICPWNOE3XNYFAOWZ3AFXAUXW4VA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.534331274 (296h41m1.131102307s) [noise=true , address=145.79.227.39:4017 , piece_id=UAN774WCTOGV5LMWAVQ2J7STHC2EP2OCH3U5AH6FP6EDSHHURFKA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk, address:145.79.227.39:4017): protocol: rpc: dial tcp 145.79.227.39:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.534583257 (296h41m1.13135429s) [noise=true , address=145.79.227.20:4441 , piece_id=XHY6L42WGOOQ3ZOB4NG3LD6XK4EHDV4VWNCCDNNPBBXZY3UK5I6A, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.534770320 (296h41m1.131541353s) [noise=true , address=145.79.227.8:4335 , piece_id=RMFE4RQFSYJENGNW77Y5CFDLEKQKR6DFHBWQZBMIL323YBYVWJSA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:1sKfG447ZZTQ6QJ1ohCxbfvm5Sv8u8azreBAqoC6EFbvXQdzwN, address:145.79.227.8:4335): protocol: rpc: dial tcp 145.79.227.8:4335: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.534886683 (296h41m1.131657717s) [noise=true , address=145.79.227.43:4158 , piece_id=WPAXDH6WSWAOICNIGVJHEKGM2SGHZYHIVWDEKOXEQSCLX5UOTZVQ, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.535239522 (296h41m1.132010555s) [noise=true , address=145.79.227.12:4770 , piece_id=UHOUUDH4JGQWJWMRR7MQIDTF4TT5X7FTACQO7SXXK3KE5RQPDQEQ, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.535256624 (296h41m1.132027647s) [noise=true , address=145.79.227.2:4113 , piece_id=4K64EHXXSVEXOGM6CZ3IANE7P5BVH2T4NOFAYMKBFDCOCCIXMPYQ, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:41:03.535651074 (296h41m1.132422107s) [noise=true , address=145.79.227.48:4304 , piece_id=CFYFXRX2L44XYVASI66OUKPJNGS67LD67HU4MZ73GP6NL6ORF3OA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG, address:145.79.227.48:4304): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:41:03.535782641 (296h41m1.132553665s) [noise=true , address=145.79.227.42:4126 , piece_id=VY2L2VREHCXEUIGES5B4XF7XMIHBFXT72XPF37LG745GPJOL3EZA, seg_pos={0 0}, upload=332303]: Done uploading piece. err: ecclient: upload failed (node:12dMWFrq2NMpnMzuPAphWerntC5kkrcnzRJZBA116uzJzyvk6fH, address:145.79.227.42:4126): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:41:03.535826211 (296h41m1.132597235s) [seg_pos={0 0}, upload=332303]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 19:41:03.561998701 (296h41m1.158769726s) [upload=332303]: Done uploading segments... Mar 18 19:41:03.562045831 (296h41m1.158816858s) [upload=332303]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.031367372s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..............................................XXXXXXXXX.XXX.] duration: 724.919479ms trace: https://jaeger.dev.storj.io/trace/6e68742e8267acf6 FAIL (2026-03-18 19:40:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 724.969738ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration: 1.477379012s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:40:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.477408818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.257397573s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.257429836s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...............................................XXXXXXXXX.XXX] duration: 5.004628895s trace: https://jaeger.dev.storj.io/trace/6e68742e8267acf6 FAIL (2026-03-18 19:40:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00473873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 1.394098006s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:39:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.394132306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 1.21208807s trace: https://jaeger.dev.storj.io/trace/168dc270f6bb5220 FAIL (2026-03-18 19:39:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.212114225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 1m0.001448867s trace: https://jaeger.dev.storj.io/trace/168dc270f6bb5220 FAIL (2026-03-18 19:39:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001470158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................................XXXXXXXXX] duration: 30.026954887s trace: https://jaeger.dev.storj.io/trace/20a925b015a683a5 FAIL (2026-03-18 19:39:50 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ad=332277]: Begin upload segment... Mar 18 19:39:20.614496056 (296h39m18.211267083s) [seg_pos={0 0}, upload=332277]: Done begin upload segment. Mar 18 19:39:20.614549497 (296h39m18.211320522s) [noise=true , address=145.79.227.2:4117 , piece_id=TLMX3EKEQ5MRSYSARBEUS4BC2SLCMEHJRGJP4Q2MADDRRNXGBBAA, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.614599156 (296h39m18.211370189s) [noise=true , address=77.90.31.16:4737 , piece_id=N4CE3LE7JCY36B3V37EGXSLH4AQESLRKMJKQEZ5JYSKK4Q3JG4VQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.615472140 (296h39m18.212243171s) [noise=true , address=45.91.116.4:4941 , piece_id=IE2LBP7KGZAOEYTCH6DLLJHATYTIN7ATAGR3LNOHTPAPRBTGOTNQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.615492434 (296h39m18.212263457s) [noise=true , address=77.90.31.18:4809 , piece_id=2MK6PVYEGE3B6QUYAQDEHQZTYNFPVA5NC7DKLEYZWPOCXIEGOPOQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.615992475 (296h39m18.212763506s) [noise=true , address=145.79.227.38:4981 , piece_id=AO32XHCNDSZIEEII6PDHOG67YOROAL4FILS3DO5EDVVWKW6F4FZQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.616084582 (296h39m18.212855615s) [noise=true , address=145.79.227.19:4081 , piece_id=JBKEGEPYIGN463LN7WL57KNJX7UZ7SXKAF3IN7EDO467HW2JYH5Q, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.616459284 (296h39m18.21323032s) [noise=true , address=145.79.227.3:4155 , piece_id=2SZH74Q25OVFJWMKANYFNQRCXOVOVQSWCX2DH45R37FVIJ62LY6A, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.616862247 (296h39m18.213633282s) [noise=true , address=145.79.227.42:4122 , piece_id=JBUM5PYVBVASQYZR4KCSZVTADJZNFRUW7DJ3K7SFCRX7B7XVINBQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.616931336 (296h39m18.21370237s) [noise=true , address=145.79.227.16:4914 , piece_id=EY7NUC72JTCGSSJ7BRHS3Z3UUWJM74DUEA72TZZNZFSEH3WPKLVQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.617388967 (296h39m18.214160003s) [noise=true , address=145.79.227.28:4731 , piece_id=DODNVXZM27KWCOUKSZKUJFXEUATGO2U5KV6BFB6RD6WNHPA76TJQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.617552728 (296h39m18.214323771s) [noise=true , address=145.79.227.32:4838 , piece_id=ZQI7O7NFI2AY5ILXZE2P6ZNTUI2RB42C6MJGVNHXYP4QRQGSCRBA, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.618201871 (296h39m18.214972904s) [noise=true , address=145.79.227.47:4266 , piece_id=HGY3XIWUAS3RAM3FEZW7LMV5Q4JI4XDTEP5HJXUYL4PEMQZRRMRQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.618326434 (296h39m18.215097469s) [noise=true , address=45.91.116.7:4019 , piece_id=SCZZNYYCDGB44CVMZRSGPCFG7VQ7Q7GE6N7I3KPHSPR643ZVQOGQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.618979236 (296h39m18.215750268s) [noise=true , address=45.91.116.8:4079 , piece_id=RFGPZB5CJMNR3YEXC2XACL6FNO52IKBW4DVLLC2DNO4TH76PJNMA, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.619768294 (296h39m18.216539329s) [noise=true , address=145.79.227.49:4340 , piece_id=5MG5BNXHLEWY2QAJF5JGQ23SKXBS63P7XZI35ED3URDKINSPDCHQ, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.620025363 (296h39m18.216796393s) [noise=true , address=145.79.227.50:4373 , piece_id=3P5JKPKT34Q6XVWVCIBL3BMHJLGC4MBZTQE7OSQNCXDC7KG57PHA, seg_pos={0 0}, upload=332277]: Uploading piece... Mar 18 19:39:20.688314400 (296h39m18.285085416s) [upload=332277]: Next returned nil segment Mar 18 19:39:20.688328989 (296h39m18.285100002s) [upload=332277]: Waiting for error group managing segments... Mar 18 19:39:20.692626390 (296h39m18.289397414s) [noise=true , address=45.91.116.4:4941 , piece_id=IE2LBP7KGZAOEYTCH6DLLJHATYTIN7ATAGR3LNOHTPAPRBTGOTNQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: <nil> Mar 18 19:39:20.693414644 (296h39m18.290185668s) [noise=true , address=45.91.116.8:4079 , piece_id=RFGPZB5CJMNR3YEXC2XACL6FNO52IKBW4DVLLC2DNO4TH76PJNMA, seg_pos={0 0}, upload=332277]: Done uploading piece. err: <nil> Mar 18 19:39:20.694099255 (296h39m18.290870281s) [noise=true , address=45.91.116.7:4019 , piece_id=SCZZNYYCDGB44CVMZRSGPCFG7VQ7Q7GE6N7I3KPHSPR643ZVQOGQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: <nil> Mar 18 19:39:20.694626076 (296h39m18.291397093s) [noise=true , address=77.90.31.18:4809 , piece_id=2MK6PVYEGE3B6QUYAQDEHQZTYNFPVA5NC7DKLEYZWPOCXIEGOPOQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: <nil> Mar 18 19:39:20.695295701 (296h39m18.292066731s) [noise=true , address=77.90.31.16:4737 , piece_id=N4CE3LE7JCY36B3V37EGXSLH4AQESLRKMJKQEZ5JYSKK4Q3JG4VQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: <nil> Mar 18 19:39:50.552503107 (296h39m48.149274139s) [noise=true , address=145.79.227.19:4081 , piece_id=JBKEGEPYIGN463LN7WL57KNJX7UZ7SXKAF3IN7EDO467HW2JYH5Q, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:145.79.227.19:4081): protocol: rpc: dial tcp 145.79.227.19:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.552928872 (296h39m48.149699905s) [noise=true , address=145.79.227.16:4914 , piece_id=EY7NUC72JTCGSSJ7BRHS3Z3UUWJM74DUEA72TZZNZFSEH3WPKLVQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:1MArPyMrdvyw4VFuArmattTtneNTaN5mt82nfGwY2sKHvQnL5D, address:145.79.227.16:4914): protocol: rpc: dial tcp 145.79.227.16:4914: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.553255476 (296h39m48.150026508s) [noise=true , address=145.79.227.2:4117 , piece_id=TLMX3EKEQ5MRSYSARBEUS4BC2SLCMEHJRGJP4Q2MADDRRNXGBBAA, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the, address:145.79.227.2:4117): protocol: rpc: dial tcp 145.79.227.2:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.553389190 (296h39m48.150160223s) [noise=true , address=145.79.227.28:4731 , piece_id=DODNVXZM27KWCOUKSZKUJFXEUATGO2U5KV6BFB6RD6WNHPA76TJQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK, address:145.79.227.28:4731): protocol: rpc: dial tcp 145.79.227.28:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.553530194 (296h39m48.150301226s) [noise=true , address=145.79.227.38:4981 , piece_id=AO32XHCNDSZIEEII6PDHOG67YOROAL4FILS3DO5EDVVWKW6F4FZQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:1u49RC6mHSy43oo9REosyNVdFB3jKk98A6hH4j9qgCQaF5qPSv, address:145.79.227.38:4981): protocol: rpc: dial tcp 145.79.227.38:4981: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.553718528 (296h39m48.150489561s) [noise=true , address=145.79.227.3:4155 , piece_id=2SZH74Q25OVFJWMKANYFNQRCXOVOVQSWCX2DH45R37FVIJ62LY6A, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:145.79.227.3:4155): protocol: rpc: dial tcp 145.79.227.3:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.553822089 (296h39m48.150593123s) [noise=true , address=145.79.227.50:4373 , piece_id=3P5JKPKT34Q6XVWVCIBL3BMHJLGC4MBZTQE7OSQNCXDC7KG57PHA, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: rpc: dial tcp 145.79.227.50:4373: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.554133355 (296h39m48.150904387s) [noise=true , address=145.79.227.42:4122 , piece_id=JBUM5PYVBVASQYZR4KCSZVTADJZNFRUW7DJ3K7SFCRX7B7XVINBQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: rpc: dial tcp 145.79.227.42:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.554363665 (296h39m48.151134682s) [noise=true , address=145.79.227.49:4340 , piece_id=5MG5BNXHLEWY2QAJF5JGQ23SKXBS63P7XZI35ED3URDKINSPDCHQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:145.79.227.49:4340): protocol: rpc: dial tcp 145.79.227.49:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.554401228 (296h39m48.151172256s) [noise=true , address=145.79.227.47:4266 , piece_id=HGY3XIWUAS3RAM3FEZW7LMV5Q4JI4XDTEP5HJXUYL4PEMQZRRMRQ, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:145.79.227.47:4266): protocol: rpc: dial tcp 145.79.227.47:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:39:50.554606245 (296h39m48.151377274s) [noise=true , address=145.79.227.32:4838 , piece_id=ZQI7O7NFI2AY5ILXZE2P6ZNTUI2RB42C6MJGVNHXYP4QRQGSCRBA, seg_pos={0 0}, upload=332277]: Done uploading piece. err: ecclient: upload failed (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6, address:145.79.227.32:4838): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:39:50.554644090 (296h39m48.151415104s) [seg_pos={0 0}, upload=332277]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Mar 18 19:39:50.577556680 (296h39m48.174327704s) [upload=332277]: Done uploading segments... Mar 18 19:39:50.577593261 (296h39m48.174364288s) [upload=332277]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.027060434s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [................................................XXXXXXXXX.XX] duration: 5.000850238s trace: https://jaeger.dev.storj.io/trace/8f1bdb0b7963128 FAIL (2026-03-18 19:39:45 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000906758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.326588336s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:39:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.326617093s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................................................XXXXXXXXX.X] duration: 5.000771271s trace: https://jaeger.dev.storj.io/trace/6c53c59bf5dd72c9 FAIL (2026-03-18 19:39:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000826856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 1.444972576s trace: https://jaeger.dev.storj.io/trace/504da329c61bc56 FAIL (2026-03-18 19:39:02 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== in upload segment... Mar 18 19:36:40.519062491 (296h36m38.115833524s) [seg_pos={0 0}, upload=332226]: Done begin upload segment. Mar 18 19:36:40.519122972 (296h36m38.115893995s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519158535 (296h36m38.115929568s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.519952691 (296h36m38.116723717s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520469473 (296h36m38.117240506s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.520531096 (296h36m38.117302138s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521025199 (296h36m38.117796226s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521195168 (296h36m38.117966191s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521517234 (296h36m38.118288257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.521721504 (296h36m38.118492529s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.522279589 (296h36m38.119050616s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523062176 (296h36m38.119833208s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.523708767 (296h36m38.12047979s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524314790 (296h36m38.121085826s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.524789568 (296h36m38.121560591s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525324320 (296h36m38.122095333s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.525779446 (296h36m38.122550472s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526254332 (296h36m38.123025355s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.526725151 (296h36m38.123496178s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Uploading piece... Mar 18 19:36:40.589597769 (296h36m38.186368792s) [upload=332226]: Next returned nil segment Mar 18 19:36:40.589617873 (296h36m38.186388903s) [upload=332226]: Waiting for error group managing segments... Mar 18 19:36:40.594936043 (296h36m38.191707073s) [noise=true , address=77.90.31.12:4595 , piece_id=ZTJWMLII2CK4LHI3TXZWZIADFIJQAD2E5ID44UEUQFRGMG3EU54A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.597084865 (296h36m38.193855887s) [noise=true , address=79.127.219.45:4876 , piece_id=3TTZKRJ5BQF6IMQ6SRTH7DK2ZX7LXJLDPYDLIBIERU4WSRCYIWFQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.610365162 (296h36m38.207136186s) [noise=true , address=77.90.31.13:4640 , piece_id=JWLWW3WC5WL3W7C774PU5EFQJJVRHGRDXTIJLIWQFLMBSV4BFSVQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.623540288 (296h36m38.220311316s) [noise=true , address=77.90.31.17:4783 , piece_id=K7YEUHBNATYTZ3AGF3PJB27L6SFI5J5PLDO3F6WTIHW45WBBOMGQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.637765186 (296h36m38.234536209s) [noise=true , address=79.127.219.46:4911 , piece_id=RGSZ7VYJJCBNEPJZ6FJ7BM6BQGOSBZQLRWXHNNB4C2C3PSN43NNA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.849102006 (296h36m38.445873019s) [noise=true , address=167.17.72.70:4756 , piece_id=RF4MNRT5B5WSDQ5OE4YIDS6FJJRKMGWEEDMPZU4GUESEDQZUYS4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:40.854370150 (296h36m38.451141177s) [noise=true , address=167.17.72.71:4784 , piece_id=ZBUUOXLEZQ353DQOY3ZSKI22H4T6VPHCDFMTQPMLGZBHYUYUXIBA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.113067742 (296h36m38.709838765s) [noise=true , address=193.149.164.67:4008 , piece_id=Y6NJEIGE5JWOBWVMTQO4BYBSY2BX7OE45TKJI3BVX7KKQYMR3Z5Q, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.117252653 (296h36m38.714023687s) [noise=true , address=79.127.205.234:4206 , piece_id=233HW3UMHJLRTISXVCO2PR3UREECB63KELEBX4ZLYGZ7JPGRGQLA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.118717458 (296h36m38.715488481s) [noise=true , address=64.49.13.137:4503 , piece_id=CGB3TCGOS6Y3UHUJS7MLV24FUDN3JVHPXQ2QIIETRONZ4H7C6S3A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:36:41.123965737 (296h36m38.720736764s) [noise=true , address=193.149.164.70:4137 , piece_id=IK2NPZKZECM55AGMLYHMGRML4RT677DLAQK2RUPNLTS2DPWZESJA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: <nil> Mar 18 19:37:40.459260224 (296h37m38.056031257s) [noise=true , address=145.79.227.37:4589 , piece_id=Z3ZPG2RWAH375JWNP3Q42MJHOG4AVZBQOAS27ISLSB7TOVFUSNHQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12ChutvH6jC7eWE7dDE9Cmsn8EDo6ecHum8dG9NBvEcdphiJgbi, address:145.79.227.37:4589): protocol: rpc: dial tcp 145.79.227.37:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.459832101 (296h37m38.056603126s) [noise=true , address=145.79.227.52:4475 , piece_id=QFVWGT7WIWXIHX3OXZKXWRZ3HNKF4EXN7C6U2OFSQMZF56QCCHQA, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460091051 (296h37m38.056862082s) [noise=true , address=145.79.227.2:4135 , piece_id=QS2HTEINCAU5IRXJHWJPZPOUXUER7LV2LA7GLPRMLXZKZM5PRPCQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: rpc: dial tcp 145.79.227.2:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460212199 (296h37m38.056983231s) [noise=true , address=145.79.227.39:4045 , piece_id=ONYBRZMCR6WZIAFOJ43XMTLVGBCH7GYTGBT5NBSY6U26WPOR72AQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1j6J2T4aX3bmjtP9J379mNQBFDp3vQZeAvmUFEYVfYZFknjrTp, address:145.79.227.39:4045): protocol: rpc: dial tcp 145.79.227.39:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460371799 (296h37m38.05714283s) [noise=true , address=145.79.227.27:4701 , piece_id=GIZEBKJF5M7QBMLYPN3WQUIJYVHHD6HXEVPGKAIIOLA273AWTPPQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:145.79.227.27:4701): protocol: rpc: dial tcp 145.79.227.27:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460512619 (296h37m38.057283649s) [noise=true , address=145.79.227.40:4062 , piece_id=BXDV657N73AWJLF54QJWYQ5Q6KTO62AKXVYVRCYKPXYNGYDH6R4A, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:145.79.227.40:4062): protocol: rpc: dial tcp 145.79.227.40:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460673235 (296h37m38.057444267s) [noise=true , address=145.79.227.36:4540 , piece_id=NCP3622NO7K3ZHBNOZVWRRCRHIQKHW5RN7N5OQCLL2UPSODMA4ZQ, seg_pos={0 0}, upload=332226]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:40.460706522 (296h37m38.057477535s) [seg_pos={0 0}, upload=332226]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Mar 18 19:37:40.478800721 (296h37m38.075571752s) [upload=332226]: Done uploading segments... Mar 18 19:37:40.478897894 (296h37m38.075668927s) [upload=332226]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1.445008618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.286427453s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:38:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.286477501s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................................XXXXXXXX] duration: 30.027276908s trace: https://jaeger.dev.storj.io/trace/1cc5b36af78717f5 FAIL (2026-03-18 19:38:55 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :25.922614217 (296h38m23.51938523s) [seg_pos={0 0}, upload=332260]: Begin upload segment... Mar 18 19:38:25.922674088 (296h38m23.519445101s) [seg_pos={0 0}, upload=332260]: Done begin upload segment. Mar 18 19:38:25.922702879 (296h38m23.519473906s) [noise=true , address=145.79.227.40:4053 , piece_id=KOYBL2ED6SSSV3O524WE4ZN635VFAFLOGNJBOFC64KBP362LPCDA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.922723646 (296h38m23.519494693s) [noise=true , address=145.79.227.10:4406 , piece_id=PANQVEE4KWHGKTCGWBXW2JBHUJ4RRCHXMBY5F6NS6EESRIQHUWYA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.923708099 (296h38m23.520479127s) [noise=true , address=145.79.227.8:4332 , piece_id=DOIBRNZVJ5OIUOMZTRNEMEWORTTLRNJ24GV4ZZ2H27V77EJT766A, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.924282179 (296h38m23.521053213s) [noise=true , address=45.91.116.13:4157 , piece_id=OLEHU77VPL47TRBOP6LYZ254QQ5UKMYXTMWWWIRBWEAQPJLSSXJA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.924583722 (296h38m23.521354748s) [noise=true , address=77.90.31.21:4882 , piece_id=VCZJ5JNCBIFBBQMH5USTANK4EGRBFOXQBDL5DFJ4V7JVUVGMZV5Q, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.924933722 (296h38m23.521704759s) [noise=true , address=145.79.227.16:4914 , piece_id=S2KC7ADOMPZ3ED7U6WZGBKER6ZQDKMFS4N76RBMQ3SCOXQWUAP4Q, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.925124238 (296h38m23.521895263s) [noise=true , address=79.127.219.45:4853 , piece_id=H5J72PLA7CYMCXPUHMXC5UE5IUHYUPX6VS6SSTQB77LW5BJTLJ3Q, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.925488229 (296h38m23.522259258s) [noise=true , address=145.79.227.13:4807 , piece_id=FW4I2DZYQV5JMWJPBSVWRRM2FSSQNSCFJJ7FVXXOLMNFOWSURPRA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.925600832 (296h38m23.522371861s) [noise=true , address=45.91.116.7:4019 , piece_id=F3INZKWHBEBLS7A3R6X5RGM5D5WALL3DX6FLNNQBIH3UGFVE5BLA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.926070601 (296h38m23.522841633s) [noise=true , address=45.91.116.4:4941 , piece_id=UXC6K3K4F2GSI4VXFICDJEZWDAFMKU52UHIVUW7F3CKYKLS2CNXQ, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.926116058 (296h38m23.522887084s) [noise=true , address=145.79.227.12:4770 , piece_id=K4FMWBDPXH4EWONYPAVWGJHYBJEEJ7UZKN7AQRVLB3QGM4JNSUQQ, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.926624185 (296h38m23.523395209s) [noise=true , address=145.79.227.30:4765 , piece_id=O2OU3VSWOAMLLMCMH6FQ2BMLPHBEAWEDUSVN3EEF4VR7TQR7ZXBQ, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.926660999 (296h38m23.523432025s) [noise=true , address=145.79.227.17:4974 , piece_id=WZR77DKY5WXGYNNR5R6CDDHVUUVO7Q7C62SDNLPZ6XW5KOLODGPQ, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.927065725 (296h38m23.523836752s) [noise=true , address=145.79.227.31:4800 , piece_id=G723MD7CVXDA2UAVT25AQWNCGL6FXMBP2ZJPC3R33EDOSCZHRDIA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.927221874 (296h38m23.523992907s) [noise=true , address=145.79.227.11:4735 , piece_id=36R7KIWH6GCMYPA42SDHC4UN3CW7MRXRIQBWEOPDEIGXNU45N3ZA, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:25.927859159 (296h38m23.524630182s) [noise=true , address=145.79.227.50:4376 , piece_id=PVKBTOYC27SJGM2SB5BFLX6LIK4DEQ4DDGY55YLHWSEYOC56DB7Q, seg_pos={0 0}, upload=332260]: Uploading piece... Mar 18 19:38:26.002923101 (296h38m23.599694156s) [upload=332260]: Next returned nil segment Mar 18 19:38:26.002941625 (296h38m23.599712666s) [upload=332260]: Waiting for error group managing segments... Mar 18 19:38:26.008041374 (296h38m23.604812429s) [noise=true , address=45.91.116.7:4019 , piece_id=F3INZKWHBEBLS7A3R6X5RGM5D5WALL3DX6FLNNQBIH3UGFVE5BLA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: <nil> Mar 18 19:38:26.010298039 (296h38m23.607069072s) [noise=true , address=45.91.116.4:4941 , piece_id=UXC6K3K4F2GSI4VXFICDJEZWDAFMKU52UHIVUW7F3CKYKLS2CNXQ, seg_pos={0 0}, upload=332260]: Done uploading piece. err: <nil> Mar 18 19:38:26.014170292 (296h38m23.610941316s) [noise=true , address=45.91.116.13:4157 , piece_id=OLEHU77VPL47TRBOP6LYZ254QQ5UKMYXTMWWWIRBWEAQPJLSSXJA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: <nil> Mar 18 19:38:26.014560547 (296h38m23.611331561s) [noise=true , address=79.127.219.45:4853 , piece_id=H5J72PLA7CYMCXPUHMXC5UE5IUHYUPX6VS6SSTQB77LW5BJTLJ3Q, seg_pos={0 0}, upload=332260]: Done uploading piece. err: <nil> Mar 18 19:38:26.014994995 (296h38m23.61176602s) [noise=true , address=77.90.31.21:4882 , piece_id=VCZJ5JNCBIFBBQMH5USTANK4EGRBFOXQBDL5DFJ4V7JVUVGMZV5Q, seg_pos={0 0}, upload=332260]: Done uploading piece. err: <nil> Mar 18 19:38:55.800497873 (296h38m53.397268899s) [noise=true , address=145.79.227.50:4376 , piece_id=PVKBTOYC27SJGM2SB5BFLX6LIK4DEQ4DDGY55YLHWSEYOC56DB7Q, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:12MnvtjSKHuNKUcVEjTviEGThRKQxzmGiMu1ofCwkv8BLAhGuzL, address:145.79.227.50:4376): protocol: rpc: dial tcp 145.79.227.50:4376: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.800683326 (296h38m53.397454359s) [noise=true , address=145.79.227.12:4770 , piece_id=K4FMWBDPXH4EWONYPAVWGJHYBJEEJ7UZKN7AQRVLB3QGM4JNSUQQ, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.800719538 (296h38m53.39749057s) [noise=true , address=145.79.227.11:4735 , piece_id=36R7KIWH6GCMYPA42SDHC4UN3CW7MRXRIQBWEOPDEIGXNU45N3ZA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS, address:145.79.227.11:4735): protocol: rpc: dial tcp 145.79.227.11:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.800989628 (296h38m53.39776066s) [noise=true , address=145.79.227.31:4800 , piece_id=G723MD7CVXDA2UAVT25AQWNCGL6FXMBP2ZJPC3R33EDOSCZHRDIA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:121yHupysREmASSVx3WUN57duu16QEs7sA2BPBWyaThFzruvch2, address:145.79.227.31:4800): protocol: rpc: dial tcp 145.79.227.31:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.801057534 (296h38m53.397828568s) [noise=true , address=145.79.227.17:4974 , piece_id=WZR77DKY5WXGYNNR5R6CDDHVUUVO7Q7C62SDNLPZ6XW5KOLODGPQ, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1, address:145.79.227.17:4974): protocol: rpc: dial tcp 145.79.227.17:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.801309805 (296h38m53.398080835s) [noise=true , address=145.79.227.40:4053 , piece_id=KOYBL2ED6SSSV3O524WE4ZN635VFAFLOGNJBOFC64KBP362LPCDA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG, address:145.79.227.40:4053): protocol: rpc: dial tcp 145.79.227.40:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.801363688 (296h38m53.398134731s) [noise=true , address=145.79.227.30:4765 , piece_id=O2OU3VSWOAMLLMCMH6FQ2BMLPHBEAWEDUSVN3EEF4VR7TQR7ZXBQ, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: rpc: dial tcp 145.79.227.30:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.801869685 (296h38m53.398640717s) [noise=true , address=145.79.227.13:4807 , piece_id=FW4I2DZYQV5JMWJPBSVWRRM2FSSQNSCFJJ7FVXXOLMNFOWSURPRA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8, address:145.79.227.13:4807): protocol: rpc: dial tcp 145.79.227.13:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.802198897 (296h38m53.39896993s) [noise=true , address=145.79.227.16:4914 , piece_id=S2KC7ADOMPZ3ED7U6WZGBKER6ZQDKMFS4N76RBMQ3SCOXQWUAP4Q, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:1MArPyMrdvyw4VFuArmattTtneNTaN5mt82nfGwY2sKHvQnL5D, address:145.79.227.16:4914): protocol: rpc: dial tcp 145.79.227.16:4914: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:55.802377256 (296h38m53.399148291s) [noise=true , address=145.79.227.8:4332 , piece_id=DOIBRNZVJ5OIUOMZTRNEMEWORTTLRNJ24GV4ZZ2H27V77EJT766A, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:38:55.802561981 (296h38m53.399333015s) [noise=true , address=145.79.227.10:4406 , piece_id=PANQVEE4KWHGKTCGWBXW2JBHUJ4RRCHXMBY5F6NS6EESRIQHUWYA, seg_pos={0 0}, upload=332260]: Done uploading piece. err: ecclient: upload failed (node:1KrucHjbKjhBZEUGd2p8cmfyCwm6rx4BVwwVhkpgnEv9EQiJRB, address:145.79.227.10:4406): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:38:55.802605769 (296h38m53.399376794s) [seg_pos={0 0}, upload=332260]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Mar 18 19:38:55.825635138 (296h38m53.422406162s) [upload=332260]: Done uploading segments... Mar 18 19:38:55.825672721 (296h38m53.422443746s) [upload=332260]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.027564882s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..................................................XXXXXXXXX.] duration: 689.37963ms trace: https://jaeger.dev.storj.io/trace/238c986c2668b78e FAIL (2026-03-18 19:38:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 689.433341ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.00400287s trace: https://jaeger.dev.storj.io/trace/355c6e2bc7955e33 FAIL (2026-03-18 19:38:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.004030525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 30.026938419s trace: https://jaeger.dev.storj.io/trace/11f402665ea287ca FAIL (2026-03-18 19:38:12 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== KFNNX7LVZRYTTSFTXYWXJ6H26UNKWRMIMHQ, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.386604076 (296h37m39.983375105s) [noise=true , address=145.79.227.27:4694 , piece_id=2HR6OYR5PLSBH74VLBZETTUTLS57XB7ISQC6K2R3KIN7QYQXC5DQ, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.387243599 (296h37m39.984014628s) [noise=true , address=145.79.227.47:4265 , piece_id=NWEKR3CFFYQWVGWCSDKF6MDAKQ7HDVTDS3OH5NXPMWKT5CDEAJEA, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.387599588 (296h37m39.984370618s) [noise=true , address=145.79.227.28:4730 , piece_id=BJBSKCKGDO4H6JJFI42NOLIUBVPF4LQ7UOWPXFQ4H7DCRCFFL3EA, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.387858882 (296h37m39.984629905s) [noise=true , address=145.79.227.29:4945 , piece_id=OHKYRK5VNWNAGM7FC3SQ7RUQTK7BMFRFIZU7YDSG25M772DOJCBQ, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.388357481 (296h37m39.985128514s) [noise=true , address=145.79.227.10:4406 , piece_id=FNOUGTLBQJP6Y3EYTF4JCSURQPZPSGWRZ3OFVIJSPGFNVS2TNG3A, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.388997600 (296h37m39.985768623s) [noise=true , address=145.79.227.50:4376 , piece_id=YPHIARVWLFITQ367EVYZSWSP3HVLMBZSQ45WLM7IGRBLYHNM36YA, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.389587027 (296h37m39.986358051s) [noise=true , address=145.79.227.49:4341 , piece_id=EDYO7KFLAORU42ANBEGTUDOA74NOC77TITMSBCXS7MJBIIUKJYZQ, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.389905443 (296h37m39.98667647s) [noise=true , address=145.79.227.38:4982 , piece_id=44FV3MVCFBJ4MAXHWPX5Y7WWUN5UCEASHJXH43SIUCZRMHK53I2Q, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.390449532 (296h37m39.987220543s) [noise=true , address=45.91.116.4:4952 , piece_id=ZHLZJYSQ2LJISUYATIW2XAVPLYAJMKLGYM3VOGXZIAPILONP6A7Q, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.391125102 (296h37m39.987896128s) [noise=true , address=145.79.227.24:4587 , piece_id=4B6Q2JJ4ATMUQT2WF2RWPSF2WZ2IPWZGNAHL7F2AONOLG5KM6YAQ, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.391142596 (296h37m39.98791362s) [noise=true , address=77.90.31.17:4773 , piece_id=LU2NJ3RQOHYKAWSDO2Y4M6MWUIVHYDEO7J5ROY3CB6A5ZMGGHMUQ, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.391676026 (296h37m39.988447055s) [noise=true , address=145.79.227.2:4113 , piece_id=QBKXFXFUBEUS55KM5AAMTEFMSMPLDECUNVUO6IQI3KDDEGCI3IKA, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.392284558 (296h37m39.989055592s) [noise=true , address=45.91.116.10:4085 , piece_id=UWJYUOZWTM72LKJWW3VZK7FMKPHYMUO4XLYJKBI45DVH4HN4U27Q, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.392948616 (296h37m39.98971964s) [noise=true , address=45.91.116.8:4047 , piece_id=D45ACB7BDVEHJJM63BTIJK7LJKYYFN4M3K7FEOHRQNSODKX2XHTA, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.393784194 (296h37m39.990555224s) [noise=true , address=145.79.227.8:4332 , piece_id=PTV7LNQYZ7FK4FL7JCFEICXZLAW7DIHBZFWV4DBFWIZTSGL4PE5Q, seg_pos={0 0}, upload=332248]: Uploading piece... Mar 18 19:37:42.467655256 (296h37m40.064426284s) [upload=332248]: Next returned nil segment Mar 18 19:37:42.467674866 (296h37m40.064445902s) [upload=332248]: Waiting for error group managing segments... Mar 18 19:37:42.471795698 (296h37m40.068566721s) [noise=true , address=45.91.116.10:4085 , piece_id=UWJYUOZWTM72LKJWW3VZK7FMKPHYMUO4XLYJKBI45DVH4HN4U27Q, seg_pos={0 0}, upload=332248]: Done uploading piece. err: <nil> Mar 18 19:37:42.471976100 (296h37m40.068747128s) [noise=true , address=45.91.116.8:4047 , piece_id=D45ACB7BDVEHJJM63BTIJK7LJKYYFN4M3K7FEOHRQNSODKX2XHTA, seg_pos={0 0}, upload=332248]: Done uploading piece. err: <nil> Mar 18 19:37:42.472533026 (296h37m40.069304049s) [noise=true , address=45.91.116.4:4952 , piece_id=ZHLZJYSQ2LJISUYATIW2XAVPLYAJMKLGYM3VOGXZIAPILONP6A7Q, seg_pos={0 0}, upload=332248]: Done uploading piece. err: <nil> Mar 18 19:37:42.472931024 (296h37m40.069702052s) [noise=true , address=77.90.31.17:4773 , piece_id=LU2NJ3RQOHYKAWSDO2Y4M6MWUIVHYDEO7J5ROY3CB6A5ZMGGHMUQ, seg_pos={0 0}, upload=332248]: Done uploading piece. err: <nil> Mar 18 19:38:12.319273170 (296h38m9.916044202s) [noise=true , address=145.79.227.2:4113 , piece_id=QBKXFXFUBEUS55KM5AAMTEFMSMPLDECUNVUO6IQI3KDDEGCI3IKA, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: rpc: dial tcp 145.79.227.2:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.319599424 (296h38m9.916370455s) [noise=true , address=145.79.227.10:4406 , piece_id=FNOUGTLBQJP6Y3EYTF4JCSURQPZPSGWRZ3OFVIJSPGFNVS2TNG3A, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1KrucHjbKjhBZEUGd2p8cmfyCwm6rx4BVwwVhkpgnEv9EQiJRB, address:145.79.227.10:4406): protocol: rpc: dial tcp 145.79.227.10:4406: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.319621880 (296h38m9.916392913s) [noise=true , address=145.79.227.52:4449 , piece_id=BIVBX6MWTV3RB3VJRKFNNX7LVZRYTTSFTXYWXJ6H26UNKWRMIMHQ, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: rpc: dial tcp 145.79.227.52:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.319933270 (296h38m9.916704302s) [noise=true , address=145.79.227.28:4730 , piece_id=BJBSKCKGDO4H6JJFI42NOLIUBVPF4LQ7UOWPXFQ4H7DCRCFFL3EA, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: rpc: dial tcp 145.79.227.28:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.320220755 (296h38m9.916991788s) [noise=true , address=145.79.227.50:4376 , piece_id=YPHIARVWLFITQ367EVYZSWSP3HVLMBZSQ45WLM7IGRBLYHNM36YA, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:12MnvtjSKHuNKUcVEjTviEGThRKQxzmGiMu1ofCwkv8BLAhGuzL, address:145.79.227.50:4376): protocol: rpc: dial tcp 145.79.227.50:4376: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.320507027 (296h38m9.917278059s) [noise=true , address=145.79.227.29:4945 , piece_id=OHKYRK5VNWNAGM7FC3SQ7RUQTK7BMFRFIZU7YDSG25M772DOJCBQ, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: rpc: dial tcp 145.79.227.29:4945: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.320585051 (296h38m9.91735608s) [noise=true , address=145.79.227.38:4982 , piece_id=44FV3MVCFBJ4MAXHWPX5Y7WWUN5UCEASHJXH43SIUCZRMHK53I2Q, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: rpc: dial tcp 145.79.227.38:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.320905920 (296h38m9.917677019s) [noise=true , address=145.79.227.47:4265 , piece_id=NWEKR3CFFYQWVGWCSDKF6MDAKQ7HDVTDS3OH5NXPMWKT5CDEAJEA, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ, address:145.79.227.47:4265): protocol: rpc: dial tcp 145.79.227.47:4265: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.321252460 (296h38m9.918023489s) [noise=true , address=145.79.227.24:4587 , piece_id=4B6Q2JJ4ATMUQT2WF2RWPSF2WZ2IPWZGNAHL7F2AONOLG5KM6YAQ, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1T4C5mqpUTmi8sZq38KuFakx6Xh6FTVBn5jCFqFHr5GAyBBDaY, address:145.79.227.24:4587): protocol: rpc: dial tcp 145.79.227.24:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.321565334 (296h38m9.918336366s) [noise=true , address=145.79.227.8:4332 , piece_id=PTV7LNQYZ7FK4FL7JCFEICXZLAW7DIHBZFWV4DBFWIZTSGL4PE5Q, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: rpc: dial tcp 145.79.227.8:4332: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.321639461 (296h38m9.918410489s) [noise=true , address=145.79.227.49:4341 , piece_id=EDYO7KFLAORU42ANBEGTUDOA74NOC77TITMSBCXS7MJBIIUKJYZQ, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:145.79.227.49:4341): protocol: rpc: dial tcp 145.79.227.49:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.321771957 (296h38m9.918542973s) [noise=true , address=145.79.227.27:4694 , piece_id=2HR6OYR5PLSBH74VLBZETTUTLS57XB7ISQC6K2R3KIN7QYQXC5DQ, seg_pos={0 0}, upload=332248]: Done uploading piece. err: ecclient: upload failed (node:17sHakGWopKRax7hbzezZASaWtH5tUUdFMP5diZDK3UwP153cd, address:145.79.227.27:4694): protocol: rpc: dial tcp 145.79.227.27:4694: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:38:12.321814751 (296h38m9.918585768s) [seg_pos={0 0}, upload=332248]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Mar 18 19:38:12.344210910 (296h38m9.940981938s) [upload=332248]: Done uploading segments... Mar 18 19:38:12.344258558 (296h38m9.941029584s) [upload=332248]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.026952959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.191848048s trace: https://jaeger.dev.storj.io/trace/15ce03727facea57 FAIL (2026-03-18 19:38:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 18 19:36:10.500432381 (296h36m8.097203414s) [upload=332219]: Starting upload Mar 18 19:36:10.500543799 (296h36m8.097314825s) [upload=332219]: Done starting upload Mar 18 19:36:10.500649790 (296h36m8.097420808s) [upload=332219]: Uploading segments... Mar 18 19:36:10.500874959 (296h36m8.097645973s) [upload=332219]: Got next segment. Inline: false Mar 18 19:36:10.500932640 (296h36m8.097703652s) [upload=332219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:36:10.500938749 (296h36m8.097709772s) [upload=332219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:36:10.530892350 (296h36m8.127663383s) [seg_pos={0 0}, upload=332219]: Begin upload segment... Mar 18 19:36:10.530984837 (296h36m8.127755871s) [seg_pos={0 0}, upload=332219]: Done begin upload segment. Mar 18 19:36:10.531072762 (296h36m8.127843796s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531740352 (296h36m8.128511384s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.531870513 (296h36m8.128641539s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532746824 (296h36m8.129517851s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.532993109 (296h36m8.129764133s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533827706 (296h36m8.130598738s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.533923098 (296h36m8.130694124s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534804584 (296h36m8.131575613s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.534867539 (296h36m8.131638564s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535728720 (296h36m8.132499754s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.535783456 (296h36m8.132554487s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536539101 (296h36m8.133310129s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.536922919 (296h36m8.133693949s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.537283766 (296h36m8.134054813s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.538845517 (296h36m8.135616541s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539332130 (296h36m8.136103158s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.539744551 (296h36m8.136515577s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.540083638 (296h36m8.136854667s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Uploading piece... Mar 18 19:36:10.601600388 (296h36m8.198371421s) [upload=332219]: Next returned nil segment Mar 18 19:36:10.601617944 (296h36m8.198388973s) [upload=332219]: Waiting for error group managing segments... Mar 18 19:36:10.606706422 (296h36m8.20347745s) [noise=true , address=77.90.31.16:4752 , piece_id=MH7M34YWITMHOLYDHZAA5HZOVME6A6NSZDUALQJRPQNAMP22PMAQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.607550275 (296h36m8.204321303s) [noise=true , address=79.127.219.45:4868 , piece_id=LU76MTXKI7UVDV5VHXLFGZO3CLWKEQJGR463E3J2LDLDB57AA3ZQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.608806703 (296h36m8.205577728s) [noise=true , address=79.127.219.38:4851 , piece_id=HAAHZMKR5BH5JBKIM2C73226FZRBVEZESYDQATE6ZJNKHH4ZPCDQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.622912404 (296h36m8.219683443s) [noise=true , address=45.91.116.14:4209 , piece_id=NUOPZWUJ7QTBQ3RWNRYC3H4I2DACNCYZ75XQYOBLUSITNU2PK3LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623232794 (296h36m8.220003819s) [noise=true , address=77.90.31.17:4798 , piece_id=PRUFHXZXFJSV456IJM4YGC6FO65S4W24K5QGXWSAJZC4H77MWYBQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.623919545 (296h36m8.220690574s) [noise=true , address=79.127.219.46:4919 , piece_id=CWWJPNSKGWZRN5JUBINAAV2TFLDTBH7DZRMV6V7SV7ZBAWROYNKQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.669238480 (296h36m8.266009504s) [noise=true , address=77.90.31.21:4903 , piece_id=JBH363RHO4FJ2G4HJIDRQM4U3H6BW2ON7DLYZLDL6WUEWKGCNZCQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.849192400 (296h36m8.44596344s) [noise=true , address=167.17.72.68:4661 , piece_id=VWCENPL2O42UCC6JOUFH337C2DDY3CYNTPCKPNGFPRWLIYE7YUCA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:10.899652863 (296h36m8.496423891s) [noise=true , address=167.17.72.69:4719 , piece_id=2AY2C35O5IKGH45UWFYDHSA6QKSCQVVIK3WCNKTNOWCCKGWQY5WA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.118324074 (296h36m8.715095097s) [noise=true , address=79.127.205.235:4272 , piece_id=4LY4OR3XG7TPI63P2Q5KVLX5LMU3XKBH4C3SMD3SNZRW3EGGJHAA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.127553350 (296h36m8.724324364s) [noise=true , address=193.149.164.69:4081 , piece_id=JORHOCGDU6SNE57F4WVH4AUVJ3HAX3V6MSB7L4YNBWKLXRC7K5LQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.145684011 (296h36m8.742455034s) [noise=true , address=193.149.164.68:4060 , piece_id=XY5RJPNM5BM27A6MJDK26WSTBHC3RJBZSRXS3VS32YUT2M7Q6UPA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:36:11.146892159 (296h36m8.743663183s) [noise=true , address=193.149.164.70:4106 , piece_id=RBCC4FVQZBNI74A4SCXY2OFDBBNMYO5FH32CPHPZDN3XOG2KEJLQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: <nil> Mar 18 19:37:10.457013911 (296h37m8.05378494s) [noise=true , address=145.79.227.52:4458 , piece_id=SH4VV6STCFPDZ4QQUKDYJKQIH6Z7X4HPFEGG4UHLAMTX7U54K53Q, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1TMPdLfCbmgi3GqDDg6AUvTCFPzbCCgwTAn9JJBrYJRnbkKtpb, address:145.79.227.52:4458): protocol: rpc: dial tcp 145.79.227.52:4458: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457080518 (296h37m8.053851548s) [noise=true , address=145.79.227.3:4156 , piece_id=E67ZTTSMDV2WBUAI4O7RUQ2QWHMA7VSGRSF32NKD4AZRP6FXKLVA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12JTrtKCNHFUY91Ktaa7v7eAWe4GHssKSeofrg3diu8isHJnmy4, address:145.79.227.3:4156): protocol: rpc: dial tcp 145.79.227.3:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.457355833 (296h37m8.054126865s) [noise=true , address=145.79.227.36:4540 , piece_id=C6JUTP5TWCMIIHQQOPCXNDTNNCXN5Y37IZQAIK6O6RHELC3ZO6AA, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1P92CRZqnx1avdDo18tMY2X7FujkZugb4gU98AaErD13gX4cLA, address:145.79.227.36:4540): protocol: rpc: dial tcp 145.79.227.36:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.458754175 (296h37m8.055525209s) [noise=true , address=145.79.227.22:4538 , piece_id=Y2MRTJ6FUF7IY6TA3RQKMTHHDJ6E62JUZ5I7K7J5D5BRR2HPPH7A, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:1kNFzPpkDtwy4VFyzeFQReorJPEVsmdDAd8ULT8ZKoGde57P3v, address:145.79.227.22:4538): protocol: rpc: dial tcp 145.79.227.22:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:37:10.459003879 (296h37m8.055774913s) [noise=true , address=145.79.227.25:4635 , piece_id=T7LMGZK2TPA5GXWJZG3OYECOPFIY2EXLYIISGOFMVX7HLUCYHPTQ, seg_pos={0 0}, upload=332219]: Done uploading piece. err: ecclient: upload failed (node:12jyQ3G4H8fYWdP6RiREP93PzZq3G9wm2BKKk2tZSfcmyopP1a2, address:145.79.227.25:4635): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:37:10.459074189 (296h37m8.055845199s) [seg_pos={0 0}, upload=332219]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:37:10.482165416 (296h37m8.078936449s) [upload=332219]: Done uploading segments... Mar 18 19:37:10.482224140 (296h37m8.078995174s) [upload=332219]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.191867606s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...................................................XXXXXXXXX] duration: 5.001477139s trace: https://jaeger.dev.storj.io/trace/238c986c2668b78e FAIL (2026-03-18 19:38:10 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001577672s core uplink us-select-1 us1-placement-20

1
...
297
298
299
300
301
...
6393