Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 462.168087ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:09:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 462.194325ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 642.170106ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:08:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 642.192439ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................XX.............................] duration: 456.167919ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:08:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.272677ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................XX............................] duration: 532.367062ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:08:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 532.452397ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 458.49127ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 458.517241ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 577.825227ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:07:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 577.883363ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 479.659784ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:07:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 479.688216ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................XX...........................] duration: 462.28826ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 463.276478ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 619.9359ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:07:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 620.023876ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 469.077216ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 469.12456ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................XX..........................] duration: 495.614877ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:06:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 495.653596ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................XX.........................] duration: 496.012869ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:06:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.083961ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 482.842455ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 482.865345ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 616.576174ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:06:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 616.606718ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 475.692543ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:06:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 475.727207ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................XX........................] duration: 477.393613ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:05:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.45247ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 500.698576ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:05:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 500.728598ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................XX.......................] duration: 479.601889ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:05:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 479.630047ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 813.946095ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:05:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 813.976778ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 498.256475ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:05:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 498.297164ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................XX......................] duration: 491.296863ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:05:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.371729ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 429.307932ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:04:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 429.335775ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 604.923168ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:04:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 604.951282ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration: 488.144757ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 488.532735ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 443.918773ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:04:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 443.96378ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 502.31224ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:04:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 502.339511ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................XX....................] duration: 458.272886ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:03:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 458.300847ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 610.368379ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:03:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 610.436224ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 469.983742ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 470.012372ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 555.207386ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:02:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 555.2393ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................XX...................] duration: 461.350302ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:02:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.415956ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................XX..................] duration: 469.51518ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:02:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.568617ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 490.654738ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:01:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 490.685346ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................XX.................] duration: 468.09ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.148268ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 605.377663ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:01:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.404211ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 493.890177ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:01:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 493.921042ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 466.737479ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:01:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 466.778438ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 639.85237ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:01:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 639.969331ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................XX...............] duration: 512.048559ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:00:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.083038ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 711.435564ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 711.45605ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 575.410876ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:00:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 575.434289ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 642.413455ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:00:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 642.474336ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................XX..............] duration: 480.01994ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:00:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 480.047585ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 429.012356ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 22:59:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 429.035681ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................XX.............] duration: 706.518147ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 22:59:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 706.542435ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 460.255039ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 22:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 460.28933ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 574.185717ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 22:59:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 574.226779ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................XX............] duration: 539.866536ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 22:58:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 539.922856ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 507.628975ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 22:58:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 507.653442ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 549.583324ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 22:58:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.62421ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)