Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [........................XXXXXXX.............................] duration:
420.04199ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:23:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 420.092062ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration:
483.183946ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:23:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 483.210241ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................XXXXX..................................] duration:
417.099289ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:23:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 417.120619ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration:
389.815928ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:23:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 389.842873ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration:
505.282613ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:23:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 505.368853ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................XXXXX.................................] duration:
444.277113ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:22:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 444.30512ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................XXXXX................................] duration:
405.599937ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:22:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 405.622123ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration:
656.79915ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:22:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 657.128675ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.........................XXXXXXX............................] duration:
393.376132ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:22:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 393.399788ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..........................XXXXXXX...........................] duration:
414.788943ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:22:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 414.812725ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration:
839.204366ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:21:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 839.30391ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration:
465.628662ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:21:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 465.650561ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................XXXXX...............................] duration:
381.918135ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:21:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 381.944384ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...........................XXXXXXX..........................] duration:
413.577045ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:21:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 413.605523ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration:
384.161449ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:21:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 384.245236ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration:
504.592719ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:21:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 504.619991ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration:
437.765547ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:20:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 437.790765ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [............................XXXXXXX.........................] duration:
411.355918ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:20:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 411.381698ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration:
406.530107ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:20:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 406.560014ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration:
512.529038ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:20:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 512.56851ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration:
367.359635ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:20:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 367.384068ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.............................XXXXXXX........................] duration:
389.901137ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:20:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 389.943802ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration:
474.559041ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:19:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 474.583945ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration:
428.868801ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:19:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 428.895238ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration:
406.60738ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:19:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 406.631861ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................XXXXX...........................] duration:
408.183385ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:19:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 408.210164ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..............................XXXXXXX.......................] duration:
420.593944ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:19:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 420.616052ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................XXXXX..........................] duration:
393.53417ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:18:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 393.559058ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration:
484.505749ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:18:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 484.594222ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...............................XXXXXXX......................] duration:
493.421885ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:18:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 493.497736ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................XXXXX.........................] duration:
486.959202ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:18:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 486.987272ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration:
355.632679ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:18:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 355.658532ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [................................XXXXXXX.....................] duration:
425.398896ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:17:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 425.42602ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration:
353.505629ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:17:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 353.529938ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................XXXXX........................] duration:
365.248136ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:17:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 365.272667ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration:
496.708976ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:17:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.73697ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration:
667.56176ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:17:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 667.584981ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................XXXXX.......................] duration:
467.531125ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:17:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 467.55702ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.................................XXXXXXX....................] duration:
500.27265ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:17:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 500.303061ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration:
505.287431ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:16:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 505.317424ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration:
404.997539ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:16:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 405.023542ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................XXXXX......................] duration:
357.067633ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:16:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 357.0933ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..................................XXXXXXX...................] duration:
384.965519ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:16:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 384.992942ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration:
366.19776ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:16:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 366.223059ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................XXXXX....................] duration:
396.211586ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:16:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 396.2359ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................XXXXX.....................] duration:
833.506717ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:15:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 833.535054ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...................................XXXXXXX..................] duration:
622.614047ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:15:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 622.639637ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration:
450.619075ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:15:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 450.635907ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration:
486.497026ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:15:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 487.035453ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................XXXXX...................] duration:
443.900596ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:15:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 443.930533ms
core
uplink
us-select-1
us1-placement-20
|