Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 562.696648ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:05:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 562.729301ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................XXX..................................] duration: 522.728245ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:05:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.760396ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................XXX.................................] duration: 581.679779ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:05:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 581.709661ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.331558486s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:04:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.33159413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 658.674528ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:04:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 658.744825ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 634.117837ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:04:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 634.151117ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................XXX................................] duration: 658.784964ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:03:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 658.813674ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.69229965s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:03:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.692344047s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 713.499507ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 713.607069ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 610.246977ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:02:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 610.281185ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................XXX...............................] duration: 576.112816ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:02:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 576.136614ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.297524651s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:02:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.297561589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.283542239s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:02:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.283654858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................XXX..............................] duration: 677.54493ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:01:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 677.575259ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 540.928823ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 541.045909ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................XXX.............................] duration: 474.821354ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:01:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.855255ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 656.528506ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:01:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 656.552285ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.322305913s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:00:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.322355923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 701.362241ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 11:00:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 701.424976ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.291063446s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.291118014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................XXX............................] duration: 479.499888ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 11:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.566227ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.265383076s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 11:00:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.265436747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 759.434605ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:59:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 759.549681ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.231814333s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:59:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.23189704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................XXX...........................] duration: 475.29558ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:59:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 475.447796ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 534.372003ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:58:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 534.552278ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.202813156s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:58:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.20284088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 532.922989ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:58:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 532.953436ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................XXX..........................] duration: 487.124778ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:58:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.162972ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.281746308s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:57:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.281775128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.539619953s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:57:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.539652988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 583.612676ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:57:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 583.653214ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................XXX.........................] duration: 428.359531ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:57:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 428.378918ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.229780679s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:56:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.229807322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 559.905322ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 559.933993ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................XXX........................] duration: 917.064239ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:56:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 917.092305ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.216741162s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:55:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.216771463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................XXX.......................] duration: 428.509838ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:55:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 428.548329ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 533.104212ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:55:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.137196ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................XXX......................] duration: 439.106ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:54:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.142008ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 556.534308ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:54:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 556.634527ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.216686559s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:54:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.216724575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.345804907s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:54:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.345836994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XXX.....................] duration: 449.065393ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:54:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 449.103946ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 668.735267ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:54:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 668.765317ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 514.588662ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:53:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 514.615272ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.264748219s trace: https://jaeger.dev.storj.io/trace/7e44fa01de290116 FAIL (2026-04-06 10:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== loading segments... Apr 6 10:39:12.293175658 (743h39m9.88994668s) [upload=832686]: Got next segment. Inline: false Apr 6 10:39:12.293210466 (743h39m9.889981489s) [upload=832686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 10:39:12.293215002 (743h39m9.889986025s) [upload=832686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 10:39:12.319174409 (743h39m9.915945433s) [seg_pos={0 0}, upload=832686]: Begin upload segment... Apr 6 10:39:12.319264769 (743h39m9.916035785s) [seg_pos={0 0}, upload=832686]: Done begin upload segment. Apr 6 10:39:12.319310748 (743h39m9.91608178s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319344730 (743h39m9.916115762s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.319839277 (743h39m9.916610309s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320290481 (743h39m9.917061503s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320450536 (743h39m9.917221562s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320733701 (743h39m9.917504727s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.320988084 (743h39m9.917759097s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321148099 (743h39m9.917919124s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321494813 (743h39m9.918265836s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321531682 (743h39m9.918302714s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.321879088 (743h39m9.918650182s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322027388 (743h39m9.918798417s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322241446 (743h39m9.919012475s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322546472 (743h39m9.919317502s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322583845 (743h39m9.919354868s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.322937532 (743h39m9.919708565s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323051576 (743h39m9.919822591s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.323474343 (743h39m9.920245373s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Uploading piece... Apr 6 10:39:12.386273803 (743h39m9.983044815s) [upload=832686]: Next returned nil segment Apr 6 10:39:12.386288014 (743h39m9.983059037s) [upload=832686]: Waiting for error group managing segments... Apr 6 10:39:12.392899406 (743h39m9.989670418s) [noise=true , address=77.90.31.17:4775 , piece_id=CFR7LFL2JZCVX4N7X2QQPJVJLZDSD6NZCNLNLSUJKU4DIZ7BBL3Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.393163291 (743h39m9.989934315s) [noise=true , address=145.79.227.41:4116 , piece_id=E43SWVMVKWDULRK4YV6SDVJBCHWSAHSFRNACXSZZZQM2IU5JJ4KA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.667307989 (743h39m10.264079022s) [noise=true , address=167.17.72.68:4665 , piece_id=H7Y37VGEKXJGRPEZ2G4NG6WXTNNJ6YQMQI4NRUZYJTWMWIVF246Q, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.714497903 (743h39m10.311268937s) [noise=true , address=167.17.72.69:4734 , piece_id=I7HV3YRWA4IH7VE2I7SQFA42VHSYJVZCKA2KP4XQ4KSRJ6S7KPLA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.734199134 (743h39m10.330970147s) [noise=true , address=167.17.72.70:4761 , piece_id=WFSQTX3PFW77TZQCSIUSHSD3LH74XTFKLYUMDNYG7MFERLUCSZHA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.896708412 (743h39m10.493479439s) [noise=true , address=193.149.164.69:4104 , piece_id=Q3VXWW73OASMNUYH77EZBCQU5GAGPG3X3TKRLED43F3R6MZYZ2DQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.899988929 (743h39m10.496759964s) [noise=true , address=193.149.164.67:4008 , piece_id=SJM4PFZHQJW7I5C3XWXIIAJF7DBXPQQYGHTB22AB7JKMG6VN7QZA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.908433760 (743h39m10.505204783s) [noise=true , address=193.149.164.70:4105 , piece_id=7CKN7FHVF6OXY6ZPFZPM4DYNDOJNHGGF3PF3VFM7V2FSTAURFGWA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.916130090 (743h39m10.512901122s) [noise=true , address=193.149.164.68:4055 , piece_id=ARLLKHKO3AQJG52QH6EWHE55IKPQIEYCIIOR4RMIMU55TGYXYGKQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:12.956247905 (743h39m10.553018928s) [noise=true , address=193.149.164.66:4985 , piece_id=A6YQDCH7PTS6CP3IFWYMQ3HJGKFX6OSYQBATKUAIOXELHAEE5WBA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.000961275 (743h39m10.597732326s) [noise=true , address=193.149.164.143:4694 , piece_id=GG3FQN4YC5QQVGFCFYWTZWOZPFQO5NL2XE4FJCSUKUCTY4L5EWMA, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.304880608 (743h39m10.90165162s) [noise=true , address=145.79.227.33:4878 , piece_id=YAEEULYYAHZP3RM6YAE55LC7S7DQLVV6JWIGYCY64GF6ITHJRQSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:39:13.307202541 (743h39m10.903973568s) [noise=true , address=145.79.227.37:4582 , piece_id=QFSMYYEUDOO5JURWQHBLBRLCOXYSP4M2OR4OBWKIZ5LCCXPXEJCQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: <nil> Apr 6 10:40:12.257884464 (743h40m9.854655494s) [noise=true , address=45.91.116.12:4257 , piece_id=GW7MFYT77HZYDY263WMUIABDAELV37MBKMBS57FFSFXCIUJ5JN7A, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ZQ5a1TwxdeuJ6UwsVm9VsWnpZEBKC6N5yYUjyNTBkriSjvYvE, address:45.91.116.12:4257): protocol: rpc: dial tcp 45.91.116.12:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258122811 (743h40m9.854893836s) [noise=true , address=45.91.116.9:4270 , piece_id=UZYXLU235ANTBCMIIQDUHM6LZUKAS64FUBJIY2NXGYTIAO776KFQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:1Vr5WmL6Gem29LQTsCwQNFuqw6j4b1nj35AgbUWDA2Lyxf43YL, address:45.91.116.9:4270): protocol: read tcp 159.223.143.39:53356->45.91.116.9:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258358693 (743h40m9.855129725s) [noise=true , address=45.91.116.6:4987 , piece_id=N2OUZHWZMNVIMAPTOHLIIT6NK5N3PVSHS6OX4ZGYSRUJIS5X3CDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao, address:45.91.116.6:4987): protocol: context deadline exceeded; context deadline exceeded Apr 6 10:40:12.258544102 (743h40m9.855315131s) [noise=true , address=45.91.116.11:4142 , piece_id=YAVLJVKDZWHWPN572Y5TX55ONWE2IDXVA6ABUF6KD7YRWCWR3YSQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:34030->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258881055 (743h40m9.855652086s) [noise=true , address=45.91.116.13:4166 , piece_id=CNVYP6YUT7E3VUCCSUR2ZWCDYU6QMOZL5GPZAMPQ5P2BVSD6HUDQ, seg_pos={0 0}, upload=832686]: Done uploading piece. err: ecclient: upload failed (node:12ajyozgSuf68kn8sUuEmuSfYCcxzU8Cwmg1EXVt9WV6dKSopy7, address:45.91.116.13:4166): protocol: write tcp 159.223.143.39:45960->45.91.116.13:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 10:40:12.258934014 (743h40m9.855705039s) [seg_pos={0 0}, upload=832686]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 10:40:12.278983295 (743h40m9.875754318s) [upload=832686]: Done uploading segments... Apr 6 10:40:12.279076909 (743h40m9.875847939s) [upload=832686]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.264880108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XXX....................] duration: 479.10483ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:53:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.1548ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 533.074906ms trace: https://jaeger.dev.storj.io/trace/7756830e434d8498 FAIL (2026-04-06 10:52:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.107556ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................XXX...................] duration: 458.046337ms trace: https://jaeger.dev.storj.io/trace/4736d76b51c361a3 FAIL (2026-04-06 10:52:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.076997ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)