Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................................XX..XXXX...................] duration:
554.349436ms
FAIL (2025-07-11 22:22:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 554.415942ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................XX...................] duration:
644.899053ms
FAIL (2025-07-11 22:22:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 644.934661ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
720.145436ms
FAIL (2025-07-11 22:22:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 720.196428ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..............................X..XX.XXXXX...................] duration:
540.873491ms
FAIL (2025-07-11 22:21:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 541.035552ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................XX..................] duration:
647.999503ms
FAIL (2025-07-11 22:21:51 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 648.029649ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................................XX..XXXX..................] duration:
528.992453ms
FAIL (2025-07-11 22:21:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 529.016989ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
667.770434ms
FAIL (2025-07-11 22:21:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 667.80422ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................XX.................] duration:
588.824387ms
FAIL (2025-07-11 22:21:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 588.853945ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
666.246637ms
FAIL (2025-07-11 22:20:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 666.268312ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................................XX..XXXX.................] duration:
549.08455ms
FAIL (2025-07-11 22:20:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.12743ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...............................X..XX.XXXXX..................] duration:
450.184772ms
FAIL (2025-07-11 22:20:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 450.210099ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [................................X..XX.XXXXX.................] duration:
489.525406ms
FAIL (2025-07-11 22:20:16 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 489.559329ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................................XX..XXXX................] duration:
554.06123ms
FAIL (2025-07-11 22:20:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 554.095624ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................XX................] duration:
707.0033ms
FAIL (2025-07-11 22:20:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 707.279241ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
627.759826ms
FAIL (2025-07-11 22:19:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 627.793736ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................................XX..XXXX...............] duration:
550.673783ms
FAIL (2025-07-11 22:19:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 550.878709ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................................XX...............] duration:
766.107874ms
FAIL (2025-07-11 22:19:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 766.139185ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
722.795118ms
FAIL (2025-07-11 22:19:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 722.853052ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.................................X..XX.XXXXX................] duration:
507.952198ms
FAIL (2025-07-11 22:19:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 507.98571ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
636.592601ms
FAIL (2025-07-11 22:19:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 636.63034ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................XX..XXXX..............] duration:
552.440899ms
FAIL (2025-07-11 22:19:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.466889ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
992.739103ms
FAIL (2025-07-11 22:18:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 992.87331ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......................................XX..XXXX.............] duration:
530.01388ms
FAIL (2025-07-11 22:18:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 530.03439ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
688.474934ms
FAIL (2025-07-11 22:18:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 688.506067ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
653.050073ms
FAIL (2025-07-11 22:18:07 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 653.081769ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..................................X..XX.XXXXX...............] duration:
1.97961528s
FAIL (2025-07-11 22:17:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 1.979647747s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........................................XX..XXXX............] duration:
497.163451ms
FAIL (2025-07-11 22:17:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 497.190939ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
636.967976ms
FAIL (2025-07-11 22:17:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 637.016887ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...................................X..XX.XXXXX..............] duration:
537.252742ms
FAIL (2025-07-11 22:17:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 537.309887ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
642.769027ms
FAIL (2025-07-11 22:16:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 642.803427ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........................................XX..XXXX...........] duration:
557.348959ms
FAIL (2025-07-11 22:16:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 557.384739ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................XX...........] duration:
538.690016ms
FAIL (2025-07-11 22:16:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 538.726581ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
641.955247ms
FAIL (2025-07-11 22:16:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 641.976915ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....................................X..XX.XXXXX.............] duration:
674.533378ms
FAIL (2025-07-11 22:16:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 674.561636ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........................................XX..XXXX..........] duration:
823.229328ms
FAIL (2025-07-11 22:15:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 823.261036ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
702.788011ms
FAIL (2025-07-11 22:15:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 702.840814ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................XX..........] duration:
548.367501ms
FAIL (2025-07-11 22:15:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 548.399481ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....................................X..XX.XXXXX............] duration:
459.433523ms
FAIL (2025-07-11 22:15:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 459.47103ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......................................X..XX.XXXXX...........] duration:
568.147917ms
FAIL (2025-07-11 22:14:54 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 568.184266ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........................................XX..XXXX.........] duration:
589.603758ms
FAIL (2025-07-11 22:14:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 589.628462ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
1.034070277s
FAIL (2025-07-11 22:14:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.034101812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................XX.........] duration:
563.807311ms
FAIL (2025-07-11 22:14:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 563.833876ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
817.117218ms
FAIL (2025-07-11 22:14:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 817.171083ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............................................XX..XXXX........] duration:
650.306454ms
FAIL (2025-07-11 22:14:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 650.331383ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......................................X..XX.XXXXX..........] duration:
539.80167ms
FAIL (2025-07-11 22:13:59 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 539.831086ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................XX........] duration:
752.643789ms
FAIL (2025-07-11 22:13:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 752.675485ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........................................X..XX.XXXXX.........] duration:
484.486818ms
FAIL (2025-07-11 22:13:43 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments...
Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload
Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload
Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments...
Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false
Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment...
Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment.
Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece...
Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment
Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments...
Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil>
Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments...
Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 484.516741ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
676.927309ms
FAIL (2025-07-11 22:13:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 676.976776ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................XX.......] duration:
814.397753ms
FAIL (2025-07-11 22:13:21 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
866s) [upload=356245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment...
Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment.
Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece...
Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment
Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments...
Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil>
Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments...
Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 814.430491ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............................................XX..XXXX.......] duration:
510.215975ms
FAIL (2025-07-11 22:13:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.240234ms
edge
gateway
us-select-1
|