Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration:
538.344314ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:40:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 538.441748ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
383.565405ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:40:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 383.592013ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration:
377.537227ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:40:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 377.568037ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration:
492.225955ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:40:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 492.256375ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [XXXXXX......................................................] duration:
375.208048ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:40:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 375.232189ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration:
507.789086ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:40:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 507.818444ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
443.056648ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:40:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 443.082485ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [XXXXXXX.....................................................] duration:
400.496481ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:39:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 400.529549ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration:
332.372274ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:39:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 332.4468ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
418.507692ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:39:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 418.536088ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.XXXXXXX....................................................] duration:
417.749831ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:39:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 417.790548ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration:
420.348829ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:39:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 420.386758ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration:
477.787474ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:39:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.810577ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [XXXX........................................................] duration:
422.921679ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:39:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 422.94733ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration:
352.467973ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:39:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 352.496321ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..XXXXXXX...................................................] duration:
426.394475ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:38:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 426.422256ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration:
447.461076ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:38:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 447.537169ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...XXXXXXX..................................................] duration:
422.504934ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:38:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 422.532021ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration:
396.115466ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:38:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 396.206427ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration:
375.689633ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:38:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 375.726273ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [....XXXXXXX.................................................] duration:
364.94834ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:38:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 364.969788ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration:
421.232156ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:38:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 421.257765ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration:
359.75644ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:38:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 359.869097ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.....XXXXXXX................................................] duration:
485.247626ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:37:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 485.270082ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration:
405.395893ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:37:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 405.42372ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration:
745.899641ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:37:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 745.945815ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration:
375.188732ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:37:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 375.215816ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration:
374.751342ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:37:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 374.797615ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration:
526.252174ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:36:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 526.301176ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [......XXXXXXX...............................................] duration:
445.983932ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:36:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 446.000693ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.......XXXXXXX..............................................] duration:
368.362278ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:36:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 368.393135ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........XXXXX..............................................] duration:
344.610232ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:36:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 344.638267ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration:
361.565441ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:36:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 361.593012ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration:
498.836198ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:35:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 498.857042ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration:
395.489061ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:35:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 395.516646ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [........XXXXXXX.............................................] duration:
419.438466ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:35:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 419.540723ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration:
358.253799ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:35:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 358.301492ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration:
525.314102ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:35:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 525.332454ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration:
426.918147ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:35:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 426.942601ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.........XXXXXXX............................................] duration:
435.750359ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:35:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 435.775975ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration:
494.201696ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:34:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 494.225611ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration:
417.927509ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:34:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 417.953121ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..........XXXXXXX...........................................] duration:
391.75549ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:34:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 391.784158ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.............XXXXX..........................................] duration:
514.001355ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:34:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 514.031861ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration:
382.300743ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:34:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 382.400866ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..............XXXXX.........................................] duration:
477.610783ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:33:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.640043ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration:
376.573516ms trace:
https://jaeger.dev.storj.io/trace/55d9d79c11454024
FAIL (2025-11-30 10:33:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
BCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181158561 (456h14m49.948841994s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181242034 (456h14m49.948925465s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181788845 (456h14m49.949472284s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.181873200 (456h14m49.949556627s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182194819 (456h14m49.949878252s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182547924 (456h14m49.950231351s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.182880110 (456h14m49.950563583s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183248855 (456h14m49.950932281s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.183669636 (456h14m49.951353064s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184107186 (456h14m49.951790611s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184606486 (456h14m49.952289921s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.184837252 (456h14m49.952520676s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185203436 (456h14m49.952886867s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.185528582 (456h14m49.95321201s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.186521941 (456h14m49.954205355s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.187568113 (456h14m49.955251544s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.188957530 (456h14m49.956640965s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Uploading piece...
Nov 30 10:01:25.252670689 (456h14m50.020354105s) [upload=474841]: Next returned nil segment
Nov 30 10:01:25.252683447 (456h14m50.020366872s) [upload=474841]: Waiting for error group managing segments...
Nov 30 10:01:25.256101522 (456h14m50.023784947s) [noise=true , address=45.91.116.13:4185 , piece_id=WONTJ2RH22J4OFJX64OCJG7XTTOEXVWWNZIXMBLZZUWZHBHBR7ZQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.256998151 (456h14m50.024681576s) [noise=true , address=45.91.116.9:4271 , piece_id=UUZDS23KRHAULFPRYXDXZX3TG5EHBOI3GRTICYKRUYI7P6W2SDHQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257115185 (456h14m50.024798612s) [noise=true , address=45.91.116.6:4995 , piece_id=R6GRYWNGRZ5PVSCFSRHX2GOXWRHWBQ4DS4BGT7CZATTBLJHXO4MA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257513315 (456h14m50.02519674s) [noise=true , address=45.91.116.14:4218 , piece_id=KBCHLKTLM7IKRRNCOOIP6MWODDA4QDPRDO3KRLMD5VIZDNBAIR6Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.257568296 (456h14m50.025251732s) [noise=true , address=45.91.116.5:4326 , piece_id=OZGENDOI7OSZ2HWM6QHLRSJAEYDDK57TTCLA4JD7YKJKAAJKADCQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.260815973 (456h14m50.028499398s) [noise=true , address=145.79.227.4:4212 , piece_id=4FU3M7MHPFN3GGKAR74WSGCCQ5CWWSNT2WTQWQV5GYZSIUUE6QSQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.270187019 (456h14m50.037870444s) [noise=true , address=145.79.227.34:4930 , piece_id=TE54XRXZISNRX45HZYGU437LQ6OJAJG6FHK5ORGL7HG4APDB43XA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.814686789 (456h14m50.582370205s) [noise=true , address=64.49.13.27:4968 , piece_id=PMIFGIH26A7VDDPLYICREHJUF32FXI23XULUMSVMOSGSO4K4BPOQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:25.845588156 (456h14m50.613271581s) [noise=true , address=64.49.13.26:4945 , piece_id=H6JOKXWT3XALRCQON5TBWLRZNA5RFAU73DH4UZIQXJP7D4FINT7A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:01:26.064651126 (456h14m50.83233454s) [noise=true , address=64.49.13.169:4714 , piece_id=GSVGXITRHXFCD2I25YNQ57O7IKF74ZFHH7YLGUNSEOZBFDVSFCHA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: <nil>
Nov 30 10:02:25.097081129 (456h15m49.864764559s) [noise=true , address=77.90.31.16:4758 , piece_id=2QTEWHXRDSTM5O6LYLW3Y5KTN3EJEQYJ3ULHE4FDRGMRVLV4IK2Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097292302 (456h15m49.864975728s) [noise=true , address=77.90.31.10:4525 , piece_id=5XWZX6U6YSF5LKAKQ36WTHJOTWFNL3AHRLNYNKCN5YFTV3V5GWXQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097472185 (456h15m49.865155616s) [noise=true , address=77.90.31.12:4620 , piece_id=TCTKX6KAARNUDOQLPD4QOUKQE7TIYZRJ6FPAGLZ4S2HMVLMDZWUA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097652471 (456h15m49.865335896s) [noise=true , address=77.90.31.17:4773 , piece_id=AZU3Z3TM4QPSDYFMT5SIKELUHKXSRRCIHFSYHG4VOSSZ5J64Q36Q, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: rpc: dial tcp 77.90.31.17:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.097830357 (456h15m49.865513787s) [noise=true , address=77.90.31.18:4824 , piece_id=E2QADXTH2XABA4NRU4DV22MMCGMNCG542BJ2U5CO3Z44QYPGOA5A, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098176306 (456h15m49.865859734s) [noise=true , address=77.90.31.19:4855 , piece_id=4HZSTQUDS3KO2H7TJJA72O52O2NB5FLTME4R4MFKDQ6SMO5OXUMQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098235197 (456h15m49.865918623s) [noise=true , address=77.90.31.15:4712 , piece_id=FMMFGTLVQQ6UCC55OLKDET54FNSME66LXKHZDBSJS4ZLWI3A7OAA, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: rpc: dial tcp 77.90.31.15:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098544044 (456h15m49.866227471s) [noise=true , address=77.90.31.21:4912 , piece_id=QSG5QTY4FSDBMGZZL763LQ5BKXEGPWLMYWOZ4OWFDO45MPDGEJFQ, seg_pos={0 0}, upload=474841]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: write tcp 159.223.143.39:38676->77.90.31.21:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.098583655 (456h15m49.866267067s) [seg_pos={0 0}, upload=474841]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Nov 30 10:02:25.120830322 (456h15m49.888513756s) [upload=474841]: Done uploading segments...
Nov 30 10:02:25.120885984 (456h15m49.888569413s) [upload=474841]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 376.612631ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...........XXXXXXX..........................................] duration:
391.834325ms trace:
https://jaeger.dev.storj.io/trace/35e31fda89825398
FAIL (2025-11-30 10:33:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 391.859625ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...............XXXXX........................................] duration:
477.745855ms trace:
https://jaeger.dev.storj.io/trace/1075f18a004aadd1
FAIL (2025-11-30 10:33:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.770548ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [............XXXXX...........................................] duration:
369.522329ms trace:
https://jaeger.dev.storj.io/trace/271f991f86732ab4
FAIL (2025-11-30 10:33:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Nov 30 10:01:25.656387382 (456h14m50.424070817s) [upload=474842]: Starting upload
Nov 30 10:01:25.656445271 (456h14m50.424128735s) [upload=474842]: Done starting upload
Nov 30 10:01:25.656560890 (456h14m50.424244307s) [upload=474842]: Uploading segments...
Nov 30 10:01:25.656755948 (456h14m50.424439364s) [upload=474842]: Got next segment. Inline: false
Nov 30 10:01:25.656798796 (456h14m50.424482221s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 30 10:01:25.656804812 (456h14m50.424488237s) [upload=474842]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 30 10:01:25.687253513 (456h14m50.454936951s) [seg_pos={0 0}, upload=474842]: Begin upload segment...
Nov 30 10:01:25.687334953 (456h14m50.45501838s) [seg_pos={0 0}, upload=474842]: Done begin upload segment.
Nov 30 10:01:25.687365999 (456h14m50.455049426s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.687395623 (456h14m50.455079046s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688017444 (456h14m50.45570086s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688162091 (456h14m50.455845523s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.688900072 (456h14m50.456583509s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.689351445 (456h14m50.457034878s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.691963311 (456h14m50.459646742s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.692615279 (456h14m50.460298707s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.693202436 (456h14m50.460885864s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695446378 (456h14m50.463129804s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.695995010 (456h14m50.463678438s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.696672685 (456h14m50.464356113s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697327512 (456h14m50.465010946s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.697946312 (456h14m50.465629743s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.698588439 (456h14m50.466271867s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.700099718 (456h14m50.467783148s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.701742223 (456h14m50.469425651s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.703455169 (456h14m50.4711386s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Uploading piece...
Nov 30 10:01:25.760996264 (456h14m50.528679675s) [upload=474842]: Next returned nil segment
Nov 30 10:01:25.761022140 (456h14m50.528705565s) [upload=474842]: Waiting for error group managing segments...
Nov 30 10:01:25.765127932 (456h14m50.532811364s) [noise=true , address=45.91.116.13:4176 , piece_id=RMHBCMZXZOZ4K7MKKWPBHLUHHS5M4KHJEURCYN267ZWEHKRTFPEQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.765342353 (456h14m50.533025792s) [noise=true , address=45.91.116.12:4230 , piece_id=MQBBHIHFRVVGE55MUD5734Y7DP35KHJYUXVIUG37H6TNLR5KZHOA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769562426 (456h14m50.537245855s) [noise=true , address=145.79.227.18:4010 , piece_id=WM6E5RW7AXH4TL67NTYDQ6DWNLF2BLRPD7FV2RFXHI3ZRZODIHSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.769651014 (456h14m50.53733444s) [noise=true , address=145.79.227.4:4202 , piece_id=YXCIJGVFQIMNTLLTRANXV2GKK5U6J6MQZSRY4ZESZ4GI2GHSF4EQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.779402836 (456h14m50.547086262s) [noise=true , address=45.91.116.5:4322 , piece_id=6TZVML2CVTFYCILQPEVTKYQUUVXB27426VFPXORXAORRODJ4VRKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.781233711 (456h14m50.548917136s) [noise=true , address=145.79.227.12:4786 , piece_id=QLZTONTBXKDDILOIUS4VEZP425CU27C4DG25VGEFUEE26646NFVQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.782297741 (456h14m50.549981166s) [noise=true , address=45.91.116.8:4057 , piece_id=DDLKYEFWA24APBF252PZC6NVJKWM43JJMQF62ION2TAXOQOKGDRA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:25.789612013 (456h14m50.557295438s) [noise=true , address=45.91.116.11:4123 , piece_id=CCP7QZFHM2QBWZTHGM5ZH6DR3YRH2DSBYJG4H2S2L5WRYX4OYVBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.366555099 (456h14m51.134238525s) [noise=true , address=64.49.13.27:4964 , piece_id=RRTDHAKO2ZITDQNMO2YKOKABSLDW3LW5ON2IGT62L4J5KWFH23VQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:26.512842120 (456h14m51.280525552s) [noise=true , address=64.49.13.154:4931 , piece_id=RNIK6MIHYPWENZJCRSMREL7CQDWZQORW3ZQ7H6XJ4VG2U3MGY5LA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:01:27.624485492 (456h14m52.392168921s) [noise=true , address=64.49.13.169:4709 , piece_id=THMOKKNHIAEOOJZSVD6XP3B2N5ZWYAUUKM2F3IINA3QNYBPBUA4A, seg_pos={0 0}, upload=474842]: Done uploading piece. err: <nil>
Nov 30 10:02:25.613199699 (456h15m50.380883132s) [noise=true , address=77.90.31.17:4786 , piece_id=ZOGFARJYPMWQRKSBI36CPDHXE2T3ZAJNNER7WI5XI6GBR7FC5GSA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh, address:77.90.31.17:4786): protocol: write tcp 159.223.143.39:42214->77.90.31.17:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613253697 (456h15m50.380937133s) [noise=true , address=77.90.31.14:4695 , piece_id=C2OMKX5PPARTCMVVOYVO6OYLO3UOGV6H5H5K7PAE2UMRXWQ2PIMQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy, address:77.90.31.14:4695): protocol: rpc: dial tcp 77.90.31.14:4695: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613513737 (456h15m50.381197173s) [noise=true , address=77.90.31.10:4548 , piece_id=G2FNV2WFZPRMGZJ7OCYLKNS5WBF6DRDKIGWD2MBA6SBOYNKB7XKQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: context deadline exceeded; context deadline exceeded
Nov 30 10:02:25.613584374 (456h15m50.381267805s) [noise=true , address=77.90.31.19:4873 , piece_id=77VCGVGDXVDOHNIG6I7SGKKNUJ4LSXWPOGXLSHGZSN6DWPBHMACA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: rpc: dial tcp 77.90.31.19:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.613872327 (456h15m50.381555764s) [noise=true , address=77.90.31.15:4713 , piece_id=2DQC2GT255JL622H72TD6TBTIKYIZF22KI6DO4HOUR56I2OZRXPQ, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614203209 (456h15m50.381886636s) [noise=true , address=77.90.31.16:4754 , piece_id=2PZWMXGP2JH5VOML6H5D2DLDBO6RXTVP7AJICZWTWRPGGJ2IOC5Q, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614515289 (456h15m50.382198723s) [noise=true , address=77.90.31.11:4564 , piece_id=NVWKGGVW2FZGL3JVESJFXQ6OBUDWC3MSNNAXF7YC7B7ZK72R3EBA, seg_pos={0 0}, upload=474842]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: rpc: dial tcp 77.90.31.11:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 30 10:02:25.614557595 (456h15m50.382241009s) [seg_pos={0 0}, upload=474842]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Nov 30 10:02:25.643219626 (456h15m50.410903052s) [upload=474842]: Done uploading segments...
Nov 30 10:02:25.643255911 (456h15m50.410939336s) [upload=474842]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 369.548782ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|