Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........XX..X.............................................] duration: 409.631861ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:38:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 409.655098ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........XX..X............................................] duration: 509.22099ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:38:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 509.253098ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 453.062112ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:38:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 453.093918ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 566.560342ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 566.606541ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.005083891s trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:37:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.005121991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 364.058779ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:37:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 364.084763ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............XX..X...........................................] duration: 415.239971ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:37:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 415.262201ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 601.762792ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:37:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 601.816053ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............XX..X..........................................] duration: 494.79157ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:36:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.839171ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 377.72862ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:36:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 377.753847ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 646.31897ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:36:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 646.359313ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............XX..X.........................................] duration: 461.643643ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:35:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 461.66715ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 562.856448ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:35:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 562.893524ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 719.506493ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 719.541109ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 535.476803ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:35:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 535.503709ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............XX..X........................................] duration: 491.059172ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:35:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.087586ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 402.04227ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:35:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 402.069083ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 596.282128ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:35:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 596.38955ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................XX..X.......................................] duration: 431.229483ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:34:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 431.259753ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................XX..X......................................] duration: 435.948617ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:34:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 436.131126ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................XX..X.....................................] duration: 457.291562ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:34:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.312771ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 473.347923ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:34:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 473.454705ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 346.376201ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:33:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 346.406844ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 937.323226ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:33:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 937.356102ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 414.924974ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:33:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 414.982013ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 487.952367ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:33:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 488.035491ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................XX..X....................................] duration: 419.548428ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:33:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 419.570804ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 622.134553ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:32:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 622.156537ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................XX..X...................................] duration: 524.544963ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:32:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 524.569015ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 509.730422ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:32:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 509.777764ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................XX..X..................................] duration: 481.98871ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:32:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.075947ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................XX..X.................................] duration: 487.736224ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:31:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.767065ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 672.200477ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:31:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 672.225007ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 475.901621ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:31:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 475.927482ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................XX..X................................] duration: 422.164517ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:30:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 422.18562ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 391.349361ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:30:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 391.385208ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 406.133185ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 406.163073ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................XX..X...............................] duration: 422.436365ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:30:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 422.544968ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 505.528102ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:30:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 505.559526ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................XX..X..............................] duration: 429.552061ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:29:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 429.58735ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 381.165997ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:29:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 381.19545ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................XX..X.............................] duration: 443.378335ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:29:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 443.404424ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 355.410588ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:29:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 355.507036ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................XX..X............................] duration: 467.015973ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:29:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 467.093691ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 419.209275ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:29:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 419.291662ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................XX..X...........................] duration: 433.367341ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 433.392455ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 452.743598ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 452.775028ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................XX..X..........................] duration: 488.160363ms trace: https://jaeger.dev.storj.io/trace/2a81d3faba8ce874 FAIL (2025-12-03 03:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.155902ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 477.909355ms trace: https://jaeger.dev.storj.io/trace/5cb2bc1add7138e7 FAIL (2025-12-03 03:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Dec 3 03:02:57.884853805 (521h16m22.652537233s) [upload=542585]: Starting upload Dec 3 03:02:57.884952100 (521h16m22.652635512s) [upload=542585]: Done starting upload Dec 3 03:02:57.885123699 (521h16m22.652807127s) [upload=542585]: Uploading segments... Dec 3 03:02:57.885367701 (521h16m22.653051131s) [upload=542585]: Got next segment. Inline: false Dec 3 03:02:57.885420704 (521h16m22.653104118s) [upload=542585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:02:57.885426360 (521h16m22.653109787s) [upload=542585]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:02:57.933581119 (521h16m22.70126456s) [seg_pos={0 0}, upload=542585]: Begin upload segment... Dec 3 03:02:57.933689239 (521h16m22.701372664s) [seg_pos={0 0}, upload=542585]: Done begin upload segment. Dec 3 03:02:57.933741183 (521h16m22.701424611s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935147010 (521h16m22.702830427s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.935437134 (521h16m22.703120571s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936544527 (521h16m22.704227963s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.936657600 (521h16m22.70434103s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937834317 (521h16m22.705517753s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.937885189 (521h16m22.705568618s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938926563 (521h16m22.706609998s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.938962578 (521h16m22.706645995s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.939995765 (521h16m22.707679194s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940282716 (521h16m22.70796615s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.940769337 (521h16m22.708452752s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941177097 (521h16m22.708860534s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.941650372 (521h16m22.709333808s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942145780 (521h16m22.709829226s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.942664689 (521h16m22.710348125s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943184171 (521h16m22.710867642s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:57.943666767 (521h16m22.711350202s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Uploading piece... Dec 3 03:02:58.016581783 (521h16m22.784265222s) [upload=542585]: Next returned nil segment Dec 3 03:02:58.016598287 (521h16m22.784281721s) [upload=542585]: Waiting for error group managing segments... Dec 3 03:02:58.021205034 (521h16m22.788888472s) [noise=true , address=77.90.31.16:4757 , piece_id=2EQO6KKCGJC5CTSNWOO3CMA5BYHFDAOWDLYP4C5RORIOMZEWM7RQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022342625 (521h16m22.790026051s) [noise=true , address=77.90.31.10:4525 , piece_id=RWRPU2IMPAFRGIEDGAOCCHFVEOTOSDRGDRM22VBSZQOKAS4OUB2Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.022401315 (521h16m22.79008474s) [noise=true , address=77.90.31.14:4672 , piece_id=FKO3IFW6EHX5PNFJEKDUUUD2Y5TH5RYKPN3JBHANUBXNNVS6GHJQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.027856890 (521h16m22.795540316s) [noise=true , address=145.79.227.35:4491 , piece_id=3XWRVZPFAMW4OQ45REUCLYUU5FH3YYMT25YJF4JH4A6NWA4ECWZA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040004133 (521h16m22.807687574s) [noise=true , address=145.79.227.33:4896 , piece_id=2UCEROHOIBUGMCRQP4JWAOOFCYDJPDZ54OSAXGEJ5DFJZQVD333Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.040098265 (521h16m22.807781695s) [noise=true , address=145.79.227.17:4002 , piece_id=QBD4HGA43LPMZ27M2MBEHPVZWQPU2FLRUQB54Y5DN7CDWDZZ53SQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.044426270 (521h16m22.812109703s) [noise=true , address=77.90.31.14:4683 , piece_id=VTRSCXJWVDUD6OYKEA72XEAHL37WN7WBFWNZUKOS4IV6C6XAFYCQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.050989511 (521h16m22.818672936s) [noise=true , address=145.79.227.44:4197 , piece_id=H6BFAEOPGLGVWRDHUDE2TMWFFNYQTFQWEN3HRAVEL7EIS3RMGMGA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.053636991 (521h16m22.821320415s) [noise=true , address=77.90.31.19:4863 , piece_id=4R73J3KULE6DPR6BPBKGUGDZ4ONVIJTQO7A2VVB7HCXJXNQBVTKQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.061073310 (521h16m22.828756735s) [noise=true , address=145.79.227.34:4930 , piece_id=FOT5TUEY4DOZDVFHCLUGSIK7QJJARAMFKT3IDWVUKA3ZHGNWY2AQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.574916475 (521h16m23.3425999s) [noise=true , address=64.49.13.27:4958 , piece_id=AHSDZBBWQ2GF6BFTGSZHY4FE3A336K7GCC2IJKEGNTBRC4L67UUA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:58.583484381 (521h16m23.351167806s) [noise=true , address=64.49.13.26:4954 , piece_id=KS5PCGLRWYJ5QEOS2GKPZVPKCITOKC4RY63ZCMCHVSAF3MHUEN6Q, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:02:59.155605485 (521h16m23.923288914s) [noise=true , address=145.79.227.15:4901 , piece_id=37PP7QWTMLAVYBPKCYOHHXWXUWHL4LYC6YTLNO6WTMENAFNF6IDA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: <nil> Dec 3 03:03:57.821771021 (521h17m22.589454455s) [noise=true , address=45.91.116.11:4142 , piece_id=IN5FO5ME6IY324GOHAVIUCEMWGWPBELQQ7RM7RHHJXUK7DUZTLYA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: write tcp 159.223.143.39:53424->45.91.116.11:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.821884257 (521h17m22.589567691s) [noise=true , address=45.91.116.4:4960 , piece_id=EGQTGEHLVYYHSHRHEY45Z36RVNGKQKEAR2OOTYVFIDLOVRJ23KZQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:1pV8Uj5R9qFy7CgtJvGACY9rAeutBCJ1WZUsZouhQN9wS4Kyr5, address:45.91.116.4:4960): protocol: context deadline exceeded; context deadline exceeded Dec 3 03:03:57.822260774 (521h17m22.589944208s) [noise=true , address=45.91.116.12:4242 , piece_id=CD2Y72UNAJYU5XA3CSEYPD252FLE6KMCG4UFLQ46TSBR3NN5B6XA, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12JsFL62jgXmza8Md8eQ9k8cyw5V9Vz2RzN4y85ZvgHZjwrq4vQ, address:45.91.116.12:4242): protocol: read tcp 159.223.143.39:36944->45.91.116.12:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822611254 (521h17m22.590294679s) [noise=true , address=45.91.116.7:4031 , piece_id=3Q7JUZCKMR4Q3NSXCE2OUHEX6SR5V2SQHZJZ4DCXHWUIKAX336XQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12aAuYPWPUFSEEXPP9VMDZPDcT5qDJd7TKNDz9esCG7RpzF83GW, address:45.91.116.7:4031): protocol: read tcp 159.223.143.39:55332->45.91.116.7:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:03:57.822770234 (521h17m22.590453663s) [noise=true , address=45.91.116.5:4303 , piece_id=XE3QWFXFHYZDF3BVDNVJ7CEYC77MMQLMQF57ZISBJIOI6GDYKIRQ, seg_pos={0 0}, upload=542585]: Done uploading piece. err: ecclient: upload failed (node:12TN4WuuSUfxS3eTfp4bacgkTzJAdtUpcKAVM6Qiad5zEvzbjMY, address:45.91.116.5:4303): protocol: write tcp 159.223.143.39:51690->45.91.116.5:4303: use of closed network connection; conn closed: op Dec 3 03:03:57.822811189 (521h17m22.5904946s) [seg_pos={0 0}, upload=542585]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:03:57.853858123 (521h17m22.621541557s) [upload=542585]: Done uploading segments... Dec 3 03:03:57.853899597 (521h17m22.621583033s) [upload=542585]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 477.937005ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 500.365574ms trace: https://jaeger.dev.storj.io/trace/6d52f5e384db403b FAIL (2025-12-03 03:27:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== t next segment. Inline: false Dec 3 03:05:02.001239017 (521h18m26.768922448s) [upload=542622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:02.001245518 (521h18m26.768928946s) [upload=542622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:02.030375194 (521h18m26.798058631s) [seg_pos={0 0}, upload=542622]: Begin upload segment... Dec 3 03:05:02.030478294 (521h18m26.798161731s) [seg_pos={0 0}, upload=542622]: Done begin upload segment. Dec 3 03:05:02.031085591 (521h18m26.798769028s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031213254 (521h18m26.798896682s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.031866122 (521h18m26.799549561s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.032483427 (521h18m26.800166851s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.033298971 (521h18m26.800982402s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035267081 (521h18m26.80295052s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.035488340 (521h18m26.803171773s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.036821239 (521h18m26.804504679s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.038421835 (521h18m26.806105271s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040148226 (521h18m26.807831663s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.040427055 (521h18m26.808110594s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041175891 (521h18m26.808859328s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.041339359 (521h18m26.809022799s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.042651899 (521h18m26.810335336s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043391194 (521h18m26.811074628s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.043953368 (521h18m26.811636803s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.044092197 (521h18m26.811775633s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.045325602 (521h18m26.813009035s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Uploading piece... Dec 3 03:05:02.200235343 (521h18m26.967918759s) [upload=542622]: Next returned nil segment Dec 3 03:05:02.200266150 (521h18m26.967949627s) [upload=542622]: Waiting for error group managing segments... Dec 3 03:05:02.217308141 (521h18m26.984991574s) [noise=true , address=145.79.227.50:4404 , piece_id=XJIX5IYCAYNZVVJ2GAJGHFZ5SHGHGCM72BZKDGCRY35LDNDLLNDQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.218748672 (521h18m26.986432106s) [noise=true , address=145.79.227.3:4163 , piece_id=O2LBI43LSOYORDVND733PTSJZMR34NCRVGMLUE36SN6FVUO2INKQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.219568197 (521h18m26.987251626s) [noise=true , address=77.90.31.11:4561 , piece_id=KDGY7RED3GFYRHQC64DWREKGZYAYSBO5OW3HFKT42Y3ZTE7BNNKA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.222900749 (521h18m26.990584179s) [noise=true , address=77.90.31.16:4750 , piece_id=YSCJKLVYOJ3XZCFT4HR2VN4YWJSPAAY6BIYWBUAPCEHISI4WPI5A, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.225169163 (521h18m26.992852597s) [noise=true , address=145.79.227.42:4141 , piece_id=D3VBSMGGIIL6XSIUGXCL3VSJFU53FM45VNTGDODCPITDUUCJINCA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.227872671 (521h18m26.995556106s) [noise=true , address=145.79.227.36:4544 , piece_id=QYFOQKCW3NT7OAJMSLHNXVYU2JIKMNNLTC5TD6LS4BWWA2C4NHDA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229051660 (521h18m26.996735088s) [noise=true , address=77.90.31.10:4531 , piece_id=KMSCBFH75CEEAKXLNUEY3EA7URYIQDQJEQGHLONCVJ257G25JN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.229172214 (521h18m26.996855647s) [noise=true , address=77.90.31.18:4826 , piece_id=2Z4IZOXH5RVPB4WUDO4CMXQ3BKLK4CYSU4MXI24P3XKAC4OU2XBA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.232409955 (521h18m27.000093389s) [noise=true , address=145.79.227.35:4505 , piece_id=RAV6XM6U2FQUIHWMMLZ3OFUGEKBY76JOFVRQWHGN2YLVM7M72SAA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246768844 (521h18m27.014452335s) [noise=true , address=77.90.31.15:4706 , piece_id=A3A6YQOY44JYI6O3HAPAWCXAX2AFYPHZ5SFWEZT5E276MX4IIXXA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.246816753 (521h18m27.014500183s) [noise=true , address=77.90.31.21:4886 , piece_id=LKQVPXR4BAXUELZKCGPLCLKSFHR3LPMPOHFTCELMYJWYZO2MVVEA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.633150681 (521h18m27.400834117s) [noise=true , address=64.49.13.154:4927 , piece_id=S57IQ3XDYDWSXYQFYO4QTZLUON6KLAA3HZNEUUVSPKQFT63CBGOA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:05:02.750233118 (521h18m27.517916546s) [noise=true , address=64.49.13.137:4499 , piece_id=5AD6OBCZJ75BI6UD5KBUQBKXV6DXKVBILNU3PYHQMMKWBN2PVVLA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: <nil> Dec 3 03:06:01.956577215 (521h19m26.724260642s) [noise=true , address=45.91.116.8:4066 , piece_id=JSA24ODSA3F6HJ4JNQYU3J6RA6AD3WYYRCBSDRDLXM5F2QFLEZWQ, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8, address:45.91.116.8:4066): protocol: write tcp 159.223.143.39:56718->45.91.116.8:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.956815514 (521h19m26.724498948s) [noise=true , address=45.91.116.4:4961 , piece_id=6HP2VQVKSX47YPAL7WQH5VMWIJ3HPUGX42OZLYD67HIVGNTEJN6Q, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957136873 (521h19m26.724820308s) [noise=true , address=45.91.116.6:4976 , piece_id=AI24KR46DVKLBSRDMKD3ZEEQD5WKSKTEFALXO77GMRV2ENNJ6UFA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, address:45.91.116.6:4976): protocol: write tcp 159.223.143.39:51580->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957373325 (521h19m26.725056768s) [noise=true , address=45.91.116.9:4290 , piece_id=RF455HAU7CWGHJB5BJFCNTZXYUXLYA4B6ZWQQIHVG2TSL2DRPCPA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:1txTbjkxQ82kwxCWNkQG4aPmVvcGARaAHxXNw1CXVTVMQFCLQ9, address:45.91.116.9:4290): protocol: read tcp 159.223.143.39:48776->45.91.116.9:4290: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957755392 (521h19m26.725438821s) [noise=true , address=45.91.116.10:4114 , piece_id=R3KVF3VZQRYUPHHKCTA6LPX3P6S45AK63PYW2Y4T7CNAZ2XGBWMA, seg_pos={0 0}, upload=542622]: Done uploading piece. err: ecclient: upload failed (node:121wUjWPiPFgX5PLRcNrs535zEL81fbPpquD9rhSaqkpWxjGZLY, address:45.91.116.10:4114): protocol: rpc: dial tcp 45.91.116.10:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:01.957803925 (521h19m26.725487356s) [seg_pos={0 0}, upload=542622]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:01.983776235 (521h19m26.751459669s) [upload=542622]: Done uploading segments... Dec 3 03:06:01.983813638 (521h19m26.751497063s) [upload=542622]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 500.389879ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)